I quit carrying 357 mag in my snub for two reasons. The 357 going through the bad guy and possibly hitting someone else is one and second is you can just about bet you could end up deaf in at least the ear on your shooting side.
A 9mm is bad enough for causing hearing loss but a 357 you just about bet you will lose a large amount of hearing. I've had loss from the 9mm and I am no dang way wanting to experience a 357.
You say you will not notice the louder 357 in the case of a shooting don't even bet on that. A 38 spl or a +P will most likely save your life and you still could have more hearing left.
As far as the baseball bat I bet the lighter bat will still take the fight out of you. I've seen a big man knocked out by a 2x4 swung by a much lighter man. Little man or not the dude was flat out for the count on the ground.