My 642 does that:stick the speed loader in and it gets stuck against the grips.

I like the factory grips,but I may need to change them because of this.