Having hunter deer for 30+ years with handgun, shotgun, muzzle loader, archery, rifle, I would say go bigger.
Yes, the 357 will kill a deer - but - for all the one shot wonders people have done there are probably 25 times as many really long track jobs or lost deer when using the 357 mag.
Treat it like an arrow and do the double lung shot, it will take it down, however there will be a track job.
Even with my 44 mag I have lost deer over the years in trying to get to fancy with the shot and ended up hitting the shoulder area and just missing the heart/lung.
Spend the time and get really good at shooting consistently, know your distance limitations.