I would have done the same thing, even though I carry at all times. It may be a minor inconvenience to leave and return later, but it might have cost you a lot more time & money if you had to defend yourself. Not to mention chance of physical and psychological injury.