You are mounted on the upper and not the handguard? Even so you'd have to be gripping like the Hulk if you were to shift the POI 3 inches.

Post some pics of the mount on the rifle so we can get a better look at potential problems with the mount. Which it does seem may be the culprit.