Probably unconventional, but I remove small rust specs with a needle and magnifying glass. If you go to the dollar stores, you can buy magnifying glasses for a buck a pair. Figure out what distance you want to be from the work (eyes 16" from the gun for example), and try out pairs until you get a set that focus at that length for you. The needle I use is inserted into a 3/8 dowel (handle). You use very very small movements and tease the rust bloom away. You can finish up with Kroil or Blue Wonder cleaner or whatever you want. You will end up with a 'bright' spot rather than a 'rust' spot.
Proceed with caution. You can cause more damage than the rust.
I will often use the side of the needle (it's not chromed, or stainless, just plain steel - actually from an old school dissection kit) to burnish very small burrs as well. If you feel around the edges (ex; trigger-guard) you can often find a 'burr' where some impact has nicked the surface. Often you can burnish the material down and make no marks on the surface at all if you use the side of the needle. For larger nicks, use the (plain steel) round side of a screwdriver (not a chromed one) to 'iron' the material that is high back down flat.