I usually wear hearing protection when hunting or just shooting. One time, while goose hunting, the seal on my electronic muffs wasn't quite right because a stocking cap I was wearing interfered with them. I didn't notice. One of my hunting buddies swung & fired his gun and there was an immediate sharp pain in my ears. For the rest of the day and the next one, it felt like I had cotton stuffed in them. I went to the hearing doc that following week and they tested me. I experienced some high frequency hearing loss.
I asked the doc why my buddies seemed to do o.k. when they didnt wear hearing protection at all. He explained that it all would depend on the angle at which they shot and how the sound traveled. (The sound carried away from the guy who was shooting, but toward my position.) Also, everyone is different in terms of how sensitive they are to loud sounds. In any case, hearing damage is accumulative over time. If you talk to many old timer shooters (or especially guys who'd been in field artillery !) you'll find that a lot of them say 'Huh?' a lot.
As for cops or soldiers not seeming to be bothered by gun blast. There is a function in the ear that seems to shut down loud sounds somewhat, as perceived by the shooter. Also, when in grave danger a person focuses on the threat. So, they may not be aware of the shot when it happens. They call it 'sensory exclusion'. Despite this, there is still permanent damage being done to your hearing.
Bottomline - - - always wear hearing protection if at all possible. After my experience with some hearing loss, I always am very careful now to wear protection and make sure I have a good seal around my ears. In addition to my gun by the nightstand, I have a pair of electronic earmuffs which I would throw on immediately if I heard 'a bump in the night'.
There may be a fast developing situation where you just do not have enough time to put on hearing protection. Because of that, I prefer to carry lower pressure rounds that use a big bullet with not a lot of muzzle blast and fuss for my defensive handgun needs. My favorites include the 45 Colt & 44 Special in revolvers with 4" barrels. In a semi auto, the 45ACP operates at lower pressures than a lot of rounds.
I have handguns in other chamberings including 357 Mag, 44 Mag, 10mm, 454 Casull etc - - - but these higher pressure rounds are always used outdoors & for hunting rather than general personal self & home defense.
Its a good idea to try to take care of your hearing. Its a lot harder to hunt, and youre not likely to be as effective in a self defense situation . . . if youre deaf.