Lot's of good comments here so, I won't double up. I do agree that you handled it OK.
One thing that hasn't been specifically said yet is, you did well to stay put. If you have moved toward him, even if it were just to hear better, that could have made him think you were being the aggressor. It's always a good idea to make the assailant come to you. It shows intent and removes any doubt as to who the aggressor is.
I also agree that you should call the sheriff. There isn't anything they can do at the moment because nothing happened. However, it gets the incident on record. Should anything happen in the future, you will have a record of past aggressive activity. This will help you should you need to defend yourself.