Have you tried it off sandbags? Have a buddy try a few shots. If he puts them in the dirt too, the gun needs to be checked out.
However, I think maybe, your flinching...
To prove it, have a buddy load 4 and leave a few empty. When you snap it on the empty chamber, you will know for sure if you react and shove the gun downwards when the hammer drops.