Depends upon what mods you're talking about. Changing sights or grips/stocks to make the weapon easier for you to shoot accurately are easily defensable. Those who suggest that mods be done by a pro are correct (OK, you don't need a pro to change stocks!). I'll also point out that S&W has a custom shop that does very good work and carries factory warranty.

Where one starts down a slippery slope is in reducing trigger pulls. If a pro reduces the trigger pull weight and stays within factory specification, this is also denfensable as it improves your ability to hit your target. If the trigger pull is less than factory specification or if you have removed safety devices, then you're engaging in behavior that can possibly be used to call your judgement into question. After all, you're potentially asking a jury to believe that your decision to use lethal force was that of a reasonable and prudent individual. In the eyes of non gun owners, who will make up the jury (and possibly quite a few gun owners, who won't be on the jury), trigger pulls less than factory spec and removed safety devices are decidedly not reasonable and prudent.

I have exactly one box stock handgun (M&P9), it flat didn't need any alteration from the way the factory made it. It's the one I've been using the past six years.

This subject has been beat beyond death on this forum and others. I recall a thread on the old Gun Zone forum that ran something like 19 pages. Take the earlier suggestion to search Massad Ayoob and/or modified guns/liability.