Damn. Is this a newer production 5906 with the Novak style Low Mount rear sight? If so, at least the tolerances might be better for both the sight base and the dovetail cut.
I remember chuckling when I once read of a comment attributed to the owner of a well known custom shop who made sights for the 3rd gen S&W's. Basically, when using a heavy ball peen hammer and a brass drift, some of the older rear sights, especially those on the older early 3rd gen's, required beating on them with a hammer like the wrath of God in order to remove them.
Having removed and installed some of them myself on the older guns, myself (as old as the late 80's/early 90's), I can't say I'm inclined to disagree.
Even after I finally decided to spend the money to order the right sight-pusher from S&W, I was told that I'd still likely have to occasionally whack the handle end of the pusher with my ball peen hammer, whenever it got too tight to turn the handle, to "break" a stuck sight loose and keep it moving. That guy wasn't kidding.
The presses used at the factory were a lot stronger than my simple 2oz or 4oz ball peen hammer and brass drifts.
Also, as has already been mentioned, some care is required to prevent losing the springs for the nylon and steel plungers during disassembly, and even more care to avoid bending, tweaking, kinking or even shearing off the "top" of the coils when installing the sights.
I remember feeling a certain amount of joy when I first encountered a .45 slide that had an oval spring plate that fit inside a recess, covering the springs, making it easy to protect the springs and keep them held down for rear sight base installation.
OP, at this point it might be prudent to have a smith familiar with the S&W guns remove the set screw and drift out the sight base. If you've already removed the manual safety assembly and firing pin, you can clean out the firing pin channel and make sure no shavings or odd bits have migrated into the firing pin or plunger channels.