That button is the striker safety. With your slide off, use your finger or a small tool and push the plunger down. Does it feel "gritty" or rough? If so, there's your roughness.

You have several options. Swap it out for an aftermarket piece, remove it and polish it, shoot it as is until it wears itself in and smoothes out, or (my personal favorite option) lube the heck out of it and then shoot it as is.