I am pretty sure that any decent load in .38 will do fine; as several have noted, the coyote is a relatively small critter in most cases. (Coyotes crossed with dogs or wolves are a different critter, but placement and bullet shape will usually prevail.) Our BostonXBoxer is the smallest adult dog we have ever had, mid-40s at the vet. She's feisty but would be in trouble with coyotes. (Answer: Rottweiler is the smallest breed I find appealing, and I am seriously considering a Boerboel in the future after I recover from my transplant.)
My default ammo in a .38 is standard velocity SWC; easy to shoot and a decent performer for almost all uses. It would be fine for this. The big issue is that coyotes in packs try to trick dogs into chasing them; unless your dog is trained to a professional level reliable recall (mine have never been), there is no place outside your fenced yard that the dog should ever be off lead. (And I mean a real lead, not a flexilead, one of the worst clown shoes devices ever invented. Watch a few vids from Garrett Wing at American Standard K9.)