Originally Posted by model3sw

Is the inner (drilled) hole the exact size of the grip studs in the frame that the grip fits snugly over it ?

If so, you need just to ream the outer recess ring withe the proper size, flat bottomed, router bit. Better to do slowly lap it by hand rather than connected to a power tool or drill press. Careful not to go too deep. If you go too deep when the screw tightens to meet the grip stud, the grip will still be loose.