Assuming the ammo in question is reliable in your gun, shot placement is most important, followed by adequate penetration. Most of the good quality self defense ammo, like HST, expands in real world shootings pretty well.

Did .40 and .45 ammo improve alongside 9mm? Yes, but not to the same degree. It's a case of diminishing returns. Real world shootings and gel results show pretty much the same performance across the board. And it's not just the FBI saying that. That's aside from the fact that regardless of skill level, most people will shoot the 9mm faster with better accuracy than .40, .45, or .357 Sig. Plus less wear and tear on guns. Plus less expensive ammo.

If you want to carry bigger bullets, fine. That's your choice. But there isn't really any need, unless you just happen to like that caliber and/or a particular type of gun.