I answer with an impatient and harried "desk four." That buys a pause. It's not what I say as much as how I say it.
Legit calls get an explanation and apology. Generally, they think it is funny.
Scammers, if they persist, get an aggressive "how did you get this number?" If they still persist...