As I have pointed out many times on various threads related to this topic, the sights on S&W M&P and 3rd Gen pistols have always been VERY tight. They are installed by a hydraulic press while the slide is held in a fixture. Front and rear are installed by the press, which pushes both sights into the slide dovetails at the same.
They are intentionally made exceedingly tight to avoid being knocked out of alignment during rough handling, dropping, etc., which is more common than most casual users might imagine with service pistols.
The need for the little set screw has always been sort of a mystery, as a proper installation in a dovetail results in the sights staying in place even if the screw is completely missing. Note that many fixed sights do not have set screws at all.
Unfortunately, many people do not understand this, and they believe that simply unloosening the screw will allow easy sight replacement. More sights and slides have been damaged by this bit of kitchen table "gunsmithing" than almost anything else apart from attempted installation of "spring kits."
Occasionally, I read of a poster who says his sight was easily removable once the screw was loosened. However, if you loosen the little set screw and your sight is easily removable (like with a drift and hammer), then you have a defective sight or slide which should go back to S&W for repair or replacement. Sights that are loose enough to allow removal that easy will drift on their own from the high speed movement of the slide or from a drop or constant smacking of the sight against the steering wheel, door jamb, etc.
The kind of forces involved in that reciprocating slide cause all sorts of havoc with sights that are only held on with the tiny screw, as even loctite will eventually give way.
Don't forget that removal of the rear sight allows access to certain moving parts - another reason for the sight to be really tight. Loctite can and has run down into the slide, and that repair job is really pricey. And don't expect S&W to fix it under warranty - they may, but if they do, it is only out of kindness, not out of legal obligation.