They are great firearms. I picked up my 500 a month or so ago - exactly the same as yours.
Had it at the range this past weekend with my chronny - all my loads were Sierra 350 or 400 gr bullets between 1400 and 1550 ftps. I tried H110, AA#9 and LilGun. All low to medium loads based on load info.

I have some Trail Boss but have yet to use that (in any cartridge). I did buy it for the 500. Seems like another range trip is in order.
These guns are plenty heavy and that absorbs quite a bit of the recoil. I put a scope on my 460XVR and that helps too by adding more weight.

If you have any time behind a 454 Casull or other big bore you won't find this too bad even with medium loads. Now, if you approach max - ymmv.