It may be just you. I know that in my case almost every handgun I own has the rear sight adjusted just a bit to the left. It all about how I hold the gun.
If the sights will adjust out to hit correctly, don't worry about it.