Never once had any problem with clothing. Belts do get some scratch marks, but even plastic clips (had those too) do that to some extent. Not a big problem if I am careful about opening up the clips before slipping over belt.
As for car seats, yes, it's a concern, I wear my IWB holster at 8:30 (left-handed), so typically only the rear clip is in contact with the seat. Both my cars have leather seats so I have to be extra careful. Even plastic clips can mar the surface. In fact, plastic could actually be worse because they typically are thicker and protrude out further than steel clips.
Usually not a problem in the colder months when there is typically outer layer to buffer the clips. In the summer months, I usually make my car seat wear an oversized T-shirt. Don't laugh... it works great. Plus I always carry fresh towels in my car. It's handy for sweaty summer driving, and to throw one behind my back to act as a buffer.
All this is for short hops around town. If I have to drive for longer distances, I remove the holster.