The length of standard punches are too long and thin, they are meant for pushing pins out of holes, and are certain to buckle under the force of trying to persuade a sight to move... they are put in with a press at the factory.
A short "starter punch" is needed to get the sight moving, then use a regular punch to drive it out. Easy to make one from one of your bend punches, just cut it off where there is only about 1/2" tip.
Better yet, buy some 1/4" brass rod at the hardware store and make your own punch to fit the need, this is what I used on my M&P 45FS to change the factory sights to a set of Warren Tactical.
I made these punches awhile back to change out the sights on a Marlin rifle.
The one that is shaped like a screw driver tip distributes the force across a wider area, instead of it being concentrated on an area the size of the pin punch, increasing its mechanical advantage.