The Shield is lightweight and thin enough that it could be called a borderline pocket gun. Sometimes pocket guns ride outside a holster.
Personally, I always carry mine in a holster and I have to recommend you do the same.
At the same time, I understand the utility of having a pistol that hangs out with your wallet, pen knife, and car keys. The reality is that true pocket guns like the BG380 and LCP sometimes go from the kitchen counter to the jacket pocket, to the glovebox, to the pants pocket, to the tackle box, and back. In my opinion, the Shield is a little big for this role, but if you ever intend to use it that way it would be totally irresponsible to do so without a manual safety engaged.
I sure wouldn't want my wife tossing one in her handbag with no safety. I've seen how much **** she keeps in there and I can just imagine an ink pen or lip balm getting inside the trigger guard.
The thumb safety on the Shield is firm enough that it is not going to actuate itself. Try one out at your LGS if you don't believe me. Also, the habit of sweeping the thumb safety off is a good one to develop. If anything slows you down in a self defense situation, it will be your own failure to recognize the threat or hesitation to respond to it, not the thumb safety.
Again, I choose not to carry without a holster, but I support your right to decide for yourself and to do so responsibly. If you think you might ever want to pocket carry, then the thumb safety is a must.