You really shouldn't have had to do that, these sights are intended to be removeable by hand. However, that spring is a bit stiff and there is a bit of a trick to removing it. First, you push it as far to the rear as it will move, then you tip it out front first. Once the front dovetail is clear of that roll pin you can relax the spring tension and simply slide it out in a forward and up direction.

Installation is the reverse, tuck the rear dovetail into the nest, compress the spring completely, then push the front down and release to lock it in place.

BTW, if you blunt the front dovetail by just 0.005 to 0.010 inch you can provide just a bit more clearance and make it much easier to change the sight out.