I have NO issues with the grips, I usually wear shirt to keep the grip away from my body, Haven't noticed it collecting fibers.
In the rare occasions I don't wear a t shirt it doesn't seem to be that bad