Well, depending on where I'm hunting and what I'm wearing usually decides the gun I carry! I don't pistol hunt, so my guns are backups to my rifles! If I am going a long distance, my rifle is usually in my backpack, which is designed to hold a rifle over my shoulder, I can get to it, but I need to reach over my shoulder and grab it by the butt, which I might have time to do, but I have found that in the woods you may not have even that much time to react. How I found this out is a story for another time! But, that said, if it is warm, I carry either my Ruger Alaskan 44mag or my S&W Backpacker IV. If it is cold, wet and nasty(basically I'm wearing a lot heavier coat/coveralls etc), I usually carry either my S&W Stealth Hunter in a shoulder/front rig (can adjust the rig with straps). Up to this point I haven't seen a need to go bigger than 44mag, but I will say that having the longer barrel is a plus for shooting say...25yds over the shorter barrels. I hunt where there is black bear, mountain lions and other critters with teeth and tusk so I wouldn't go any smaller! I have considered taking a 460..not because I need bigger, but because I LOVE the 460 round, but the noise!!! I would have to wear electronic hearing protection because it is painful to be anywhere in the vicinity of that round going off! But I do shoot them at near max, so maybe if I "toned it down" to say somewhere under 2000fps it would make the gun more viable...that or use 454's....but haven't gone there yet!
Bottom line though, if you reload, load to the recoil and accuracy that you are comfortable with. Might want to consider firing a shot or two off out doors without your ears on as well....