I've had and still have plenty of guns that have loose tolerances, especially in the slide to frame fit and they will still produce much tighter groups all while being 100% reliable.
I have P229 .357 that is like that. It has been shot - a bunch. You can hold it in your hand and shake it and the slide will rattle around like pennies in a coffee can.

It probably shoots into about 2-inches at 25-yards with common Remington 125-gr FMJ ammunition.
I don't know if lack of tight fitting is the issue with the M&P9, nor do I think it is related to twist rate. I am not sure how one could say the slower twist accounts for about an inch of lost accuracy. I'd like to know more about that.
When one speaks of fitting, even tightly fitted pistols (like a Les Baer .45) are not "tight" in the sense that a toolmaker is used to. Any gun has to have some room to run, to get dirty, to heat up and cool down, etc.
My Premier II may have about 0.003" clearance at the hood. I have a 1911 Pro 9mm that has about 0.008" there. Other barrel fitting measurements are similar between the two guns. The Baer never throws a wild shot. The Pro usually throws one round wild in a 5-shot string, sometimes two. (Not necessarily the first or last shots.) Sometimes the Pro will shoot one or two nice groups, and then it will go back to throwing a wild shot. When it does throw one out, it is consistently high and a bit left.
Is that wild shot a matter of the hood fit? It certainly could be. Would I be willing to bet a small some of money on it? Probably not. You might fix that and find you still have a problem. It could be a combination of any of a dozen or more other things.
Pistols are simple things when you are not expecting too much from them. When you start to raise the bar, they suddenly appear to become a bit more complicated. At my age and with my vision, could I get along with a pistol that shoots into 3-inches at 45-feet? Yes, but that is not what I really want on my belt. What I want is 1-inch per 10-yards. (And of course I also want to be able to shoot that well every day I pick up that pistol and ammunition.)
You won't win a bullseye match shooting like that, but you won't go home in disgrace, either. 4-inches at 25-yards is a long way from that. To expect 4-inches from any quality, duty-size handgun seems pretty reasonable to me. I've had dozens of guns that would do that. If that is more than you need, that's OK - I am glad you are happy! All I am trying to say is, in my opinion, if the pistol builders are doing a really good job, that 4-inches/25-yards thing is not at all out of line. I'd take that in a duty grade gun and not complain too much, but I am looking for more. Less is not really an option.