Not to beat a dead horse, but I originally carried without one in the chamber. I personally don't think that the time in racking the slide would be that huge (and I practiced racking it as I drew it).
But a post here made me switch my behavior. The point being what if someone has already griped my left arm. Or as posted earlier here, the arm is already occupied somehow. Since then I always have one in the chamber.
Honestly, as long as it's in the holster, it's safe. Chambered or not. Just need to be aware as you put it in the holser.