From a truly technical aspect, results are skewed if you exclude high and low velocities and flyers. If you have a truly accurate load, shot-to-shot deviation (unless it's really wild) means little unless you're shooting at very long range. I wouldn't be concerned with "normal" deviation, particularly with handgun ammunition.

"Flyers" are different and more important. If you can shoot enough groups and eliminate the Ransom Rest stability or a problem with the grip inserts, then something is causing flyers. However, it depends on how much trouble you want to go to in addressing the problem. Lube purging? Might be, but isolating that as a cause would require additional group shooting.