Removing factory installed sights is usually very, very difficult. The factory apparently will not take the time to fit sights, so they just ram them home with some kind of hydraulic pusher, I assume. They are not just tight; they are smashed in there in many cases.
Trying to use a punch and finding out it won't work usually involves damage to the sight, slide and/or tool. Its just not worth even trying IMO.
A sight pusher, a good one, makes all the difference. A cheap one will have the same liabilities as a punch. If you go to a gunsmith, ask if he uses a good pusher. If he doesn't, find another GS.
When you install new sights, you can fit them so they will be secure but can be moved with a punch, but if you already have access to a good pusher, use it.