I personally think that it would be hardest on the hole in the recoil shield, then on the center pin and the center hole in ratchet. The yoke to frame coming together has a lot of mass and area to take up the shock. The hole in recoil shield will have its off side struck the same way every time you flip it closed. The center pin and ratchet hole will have the forces spread out 360ᵒ depending on where the cylinder is each time. The tube of the yoke with be loaded a bit by the cylinder with the same directional forces every time also, with the unsupported end by ratchet getting the most. While I don't think that a few times or even 100 times will do much eventually it will add up. The larger and heavier the cylinder the sooner it will be apparent. An N frame 357 probably the worst for this as it would slam the center pin into the latch hole with the most mass.
Take a small 4 oz ball peen hammer and start lightly tapping on something similar to a front sight. Nothing at first, but with enough strikes the top it will slowly mushroom. Keep flipping that little 38 and eventually that hole in the recoil shield will be egged out and the one in the ratchet will enlarge a tiny bit. 500, 1000 flips, 5,000. 10,000 to cause enough to effect accuracy, I don't know, I do know it WILL happen.
Take one and fire a group from a Ransom rest, flip it 100 times, fire it from rest again, repeat. Probably take some time to see anything on the targets. Be an interesting experiment for YOUR gun. Not mine. LOL
I not saying a flip or two is trashes a gun. Hardly. But it will slowly cause un needed damage.