I carry my full size M&P9 Pro on- and off duty, hot/cold/windy/rainy don't matter, don't care. Do I have smaller guns? Yes. But I'm not trusting them with my life. My 9mm M&P? All day, every day.
Bottom line, like others have already said, carry what you are most comfortable and proficient with. Use a good belt/holster and it won't bother you. Or at least not as much.