No, the pin should not be "walking" out after 4 or 5 pulls on bolt. It is something you could fix yourself if you are so inclined or either take it back to the store or call S&W customer service.
Welcome to the forum, you can learn a lot about your Sport here.