Just my experience,
Have the cylinder charge holes chamfered, its a small chore easily handled by any decent gunsmith. It just involves barely breaking the edge of the cylinder holes and makes it much easier.
The grips ARE important as others have mentioned. The factory ones look 'okay'.
The speedloader's you chose are important. Safarilands are pretty much the hands down winner. HKS aren't bad once you get the hang of them.
The main thing to remember is, GRAVITY IS YOUR FRIEND. It takes a bit of practice, but do your best to have your barrel pointed downward as much as you can. This goes double for the HKS speedloaders. As others have said, once you feel the bullets have started just twist the button and drop the loader.
Happy shootin',
Murphy