Won several competitions doing just that with my M586 -5. 3.5gr Bullseye under any 158gr LSWC.
Try putting the same load in 357mag cases and see how they do. Make sure you clean out any carbon rings first though.
That will tell you if it is the case length or not.
Are your bullets super hard? Maybe they aren't big enough or soft enough to obturate, hence the accuracy issue.
If no better with longer cases, go back to special ones, bump the load up to maximum and see what happens, then go to 357mag cases again, work up to maximum for 357mag and see if you get any better accuracy.
Using faster powders for harder bullets may help because to get the same velocity as slower powders, the pressure is higher. That makes the bullet "squish" better and fit the lands and grooves better.
Just a thought.