Spegels work well with speedloaders. If you prefer rubber, find a set of the older Uncle Mike's factory J-frame boot grips. These were designed by Spegel and also provide enough clearance for speedloaders. The newer factory rubber grip looks similar but is a bit thicker and doesn't work well with speedloaders, as you folks are finding out.

The grip on the far right here is the one you want to look for (note how deeper the relief is):