Knocking off the sharp corners and cleaning up burrs on the flat side and bottom (if there are any) is all that is necessary. Many owners who use their firearms for competition also lightly polish the interior of the spring recess. My instructors at the armorer's school also suggested checking for burrs and de-burring, if necessary, the sides of the tongue at the front of the rebound slide. Creating a mirror surface you can see yourself in is a waste of time IMO.
Some use stones for this purpose, like Arkansas, or a 6" medium or fine India (aluminum oxide) stone, similar to what the factory provides in the armorer's tool kits. Others clean up these flats and edges with very fine wet/dry paper on a flat surface, like a piece of glass, for example.
To answer your question, yes, it is possible to damage internal parts, including the rebound slide, by excessive stoning and polishing. Even very small alterations in dimensions can adversely affect a particular part's fitness and it's ability to function as designed.