It could be the grips, but I doubt it. J frame's tolerances are just too tight for a lightning fast speedloader reload. What I would suggest is lose the HKS loader and switch to a Safariland Comp I loader.

With the Safariland, you can insert a couple of rounds and turn the cylinder with your fingers while pushing on the loader. Safariland speedloaders release the rounds by the little device in the middle when it comes in contact with the cylinder's star instead of a knob like the HKS.

Once enough rounds are in the charge holes and the loader is satisfied with it's chambering, it will jetison the cartridges by itself. You can then close the cylinder and the loader will fall away from the gun, putting you back into the game/fight.