Hmmm, that's interesting. Most guys complain that the safety is always way to stiff and want to know how to loosen it.
Anyway, remove the grips, remove the screw and bend the safety spring metal detent in a bit to tighten it up. Reinstall and your problem should be solved.
Another suggestion is to hit the sharp edge of the safety switch with a very fine Arkansas stone once or twice. This will take off the very sharp edge and should help prevent it from catching on your gloves.