Before removing material, you might disassemble/remove the safety to confirm the proper plunger and spring have been used inside the safety body itself. If the wrong spring is used, the plunger will not exert the proper tension on the assembly to prohibit it from lateral movement in the slide cavity.
The problem could also be related to the interface of the removable, right-side lever in its slot in the safety body. The right side lever should fit the slot tightly when inserted into it's "T" slot. If there is too much open gauge between the lever and the slot in the cavity, the lever itself can be moved sufficiently to impact the frame when the safety is engaged. Carefully peening the lever stem on each side can help close the gauge and keep the lever from tipping down. (If this is determined to be the issue) Side to side gauge on the frame rails (normal wear and tear) can also contribute to this issue.
