Depends some on the situation...
The smallest caliber I carry for SD is 38 special or .380 ACP which are about the same as far as knock down and stopping power. This is when time and clothing does not permit something like a 9mm or .40 S&W or .45 ACP.
For me ease of carry is important to make certain I always have a gun with me. That is more important to me than carrying a lager caliber.
If I knew for certain that I would need a gun in advance, I would probably be carrying a 4013TSW or a 4513TSW as back up to a shotgun. But since I have carried for several decades and never needed to pull a gun, I consider carrying a lot of extra weight or wearing a jacket to be more trouble that I am willing to deal with, just to have a larger caliber.
I could see how some might even carry a .32 ACP if they were highly sensitive to recoil. I can see a .32 for an elderly woman or man with little ability to take recoil.
I understand that larger calibers are typically more effective it all other things are equal, but to me what is most important is a gun that I will not leave behind and that I am able to control or handle effectively.
I have never felt uneasy with a snub nose .38 special in my pocket. However if traveling or going into areas that are know high crime areas, I typically carry something a little more powerful and with more rounds.