Don't use anything abrasive,,even the slightest bit abrasive, on case colors. You'll wear them right off.
You can lighten them up and take them off with something as common as an eraser.
It's used by engravers to give the 'gray' background effect sometimes around a figure or inlay inside of a border on a case colored or sometimes even a blued finish.
Much quicker and easier than messing around with a chemical to do the smaller tedious jobs.
Also chemical that would remove bluing,,will also remove case colors.
Any of the rust remover chemicals (Naval Jelly, ect) will almost instantly take them off.
The surface will still be hardened (case hardened) if in fact it was during the process,,but the colors will be gone.
Any of the home brew bluing removers & some bug sprays have a chemical that'll remove blueing & colors for you,,
Most case color jobs are coated with a clear or tinted coating after they are done. Anything from shellac (orange shellac was the most common original coating), laquer, a thin wiped on coating of most any decent stock poly finish, ect.
Oil will generally not remove the colors but can attract grime that becomes abrasive that will.
Another good reason for a thin protective coating on the metal.
They put some strange stuff in the bore solvents these days and not knowing what's in them, I'd keep them clear of the nice colors unless I knew for sure.
Case colors are just a fragile finish. No getting around it. Some will hold up better than others. Some will wear with little handling.
Most wear quicker than hot blued surfaces but since they are of a mottled look to begin with, general fading and wear isn't as noticable than patches of worn off bluing exposing grey steel underneath.
