It's not necessary to sweep the muzzle across the person next to you unless your intention is to draw your pistol with as much flourish as possible. Until it is being raised toward your target the muzzle should always be pointed down. However if the range you practice at will not allow it then it's a good reason for not wearing it at the range.