I feel that way about all semi-autos with aluminum frames, not just S&W. I will still consider an aluminum frame revolver even though I do not currently own one. And I do own a S&W 22 compact which has an aluminum slide on a polymer frame. But its not just S&W that has had issues with aluminum frames, I have heard the same about Sigs too.

If S&W made a steel frame metal version of the M&P compact I would likely buy it as a range and home defense gun. It would be too heavy to carry but the extra weight and lack of flex in the frame would make it easy to shoot well. But I have no interest in the current metal model.