It makes me freaking crazy! I purpose myself as a good listener and try to respond with an intelligent answer if I know it.
I have no problem with saying I dont know. Please don't butt in while I'm trying to speak.

End result is an intellegent conversation.