No real downside I can see. At least with K-frames, a 3" barrel seems to give a good balance between ballistic performance, handling, and carry-ability. I personally think it's the most aesthetically pleasing combination. I used to have a 3" 65 and it was a great carry gun.
Not a fan of 3" J-frames, though. For me, the J-frame's biggest advantage is carry versatility (IWB/OWB/AIWB, pocket, ankle, shoulder, etc.), and that means a 2" barrel. I'm sure others will have differing opinions.
Don't really have an interest in revolvers larger than K-frames, so I can't really comment on those. I do have an interest in the 3" Ruger GP100 with fixed sights, but that's mostly because of its role as the standard issue sidearm for the US Marshals Service before the Glock 22.