On all the ones I've changed sights on (4), the rear sight is much easier to move than the front. My sight pusher wouldn't budge the front sights, so I used a brass drift.
If you have to move the rear sight very much, you might want to split the difference and move the front sight the opposite way.
That way your rear sight isn't as far off to the side.

__________________
U. S. Army Veteran