I haven't noticed a difference between 165 & 180. Admittedly all of my range ammo is 165 gr so I shoot that the most but my SD ammo is mostly 180 gr & I don't shoot much of it. Just enough to cycle the old stuff.
I've tried both in a back to back range session (1-box 165gr & 1-box 180gr) with my M&P 40, compared the groups, and found they were a bit more accurate with 165gr.