We had a case this summer where a Home Invader beat and then stole the car of his victim. That victim naturally being a bit PO'd grabbed his handgun and shot at the retreating criminal from his front yard. He missed and one of those misses traveled about 200 feet, passed thru a layer of wood siding, the cladding, and the interior lathe and plaster and killed a neighbor preparing her dinner. He has since pled out to, I believe, Felony Manslaughter.
Point is, you are responsible for every single miss, whether it hits a family member or someone down the street you've never met. Use the caliber you are MOST familiar with and that you shoot best with.
Personally, since I don't reload, I don't do much shooting with 357 Magnums, it's just too expensive. However, I have shot with it enough to know that it has a VERY detrimental effect on my firing rate. Compared to 38 spl. or 38 +P, my firing rate has to be cut by 50 to 60% to get good hits with the 357 Magnum. BTW, that's with a 38 ounce model 620, in a lighter gun such as a model 60 my firing rate would probably slow down to 2 or 3 SECONDS per shot with the 357 Magnum. IMO, that isn't acceptable, especially considering that I can run 38 spl. into a sub 5 inch group at 30 feet at a 1/2 second interval with the 620.
Note, my experience is that old saw "practice with the 38 and use the 357 Magnum" is really TERRIBLE advice. It doesn't matter what you're shooting, missing is BAD NEWS. If you don't practice with the 357 Magnum, you WILL miss with it in Combat. The POI changes markedly and the difference in recoil has to be felt to be understood. The simple truth is the 357 Magnum is a caliber that is difficult to master, is a real "flinch builder", and requires a lot of practice to maintain proficiency with.
PS; while I'm pretty good with a 38, my carry choice is a 40 caliber Sig P239, that one I am scary good with at 40 feet or less. Because the 40 caliber is relatively cheap to practice with, the P239 gets shot a lot, more than any other handgun I own. So, I am not unfamiliar with somewhat snappy recoil, however my response to shooting a 357 Magnum the very first time was "holy smokes, that hurt". Since then I've learned to tolerate shooting the 357 Magnums but I don't think I'll ever master it to my satisfaction. So my choice for HD is the same as what I carry every day, my P239. Because it's the one gun I know I won't miss with even if the "target" is shooting back.