It sounds as if you are anticipating recoil and have developed a flinch.
If the gun groups well with .38's it should do the same with .357 loads.
Next time you try to shoot the .357's leave one chamber empty and spin and close the cylinder. See what happens when you get to the empty chamber. I'll bet the gun goes down and left if you are right handed.
There is a target picture posted somewhere which lists all the common shooter errors. I'm sure you will see it soon.

Bruce

Check this previous thread for the target info.
Shooting low left