Exactly. As I understand it you (the gun carrier) can be charged with either menacing or inducing panic, depending on the level of the Hissy-Fit the scared person exhibits.
And in this day and age where folks seem to go out of their way to become offended, as well as the 15 minutes of fame that goes along with the "Instant Reality Show" of seeing their mug on the TV, it could get really, really ugly!
I don't worry too much about the lump my gun may add to a pocket or baggy sweater but I am very careful about raising my arms to get something off a shelf, bending over to tie a shoe, etc. as I do not want any part of the gun itself showing. When I get a new holster I wear it around the house to make certain I am properly "dressing around the gun."