My stock SD40VE does the same thing.
It has never posed a problem as far as functionality is concerned, so I just live with it and pop it back in when it pops out...

One great big festering neon distraction