If you integrate thumbing off the safety with your draw/presentation there is zero delay attributable to that action since the safety will be off before the gun reaches aim point. You simply switch it into fire position as the barrel rises from pointing down at the floor (when drawn from an IWB, OWB or pocket position.) Hence the time involved should not be a factor no matter how quickly a threat presents itself.
In a decade around law enforcement I had knowledge of probably 50 or more cases where police officers fired their weapons at suspects/attackers. I can only recall one case (which I witnessed personally) where there was an immediate draw/fire against a suspect who was also drawing a weapon. In the vast majority of cases there is some warning, especially for a situationally aware officer who sees hands in pockets, reaching under a car seat, etc. In a fairly large number of those cases the officer(s) already had their weapons in hand as they approached the situation, often alongside the seam of the trousers. There just really aren't many "high noon/draw pardner!" situations out there.
Since civilians are likely to face similar situations (carjackers approaching at the gas station, muggers sizing you up on the street) any reasonably alert CCW holder should already have a hand on the gun butt and the safety off before the need to present and possibly fire the weapon.
I carry either a double action hammerless LCR .357 or a .45 Shield, the latter with thumb safety engaged. I never feel unsafe because of that and in fact enjoy an elevated sense of safety.
By the way during that career I knew of at least four officers who experienced negligent discharges with revolvers, in all four when holstering weapons where something on the equipment belt engaged the trigger. I also saw several what I would term accidental discharges during fights and in one case a foot pursuit where the officer went to the ground tackling the suspect.
I am comfortable with a thumb safety and have never considered it an impediment to any necessary self defense reaction.