Not knowing whether upper or lower has the tolerance issue, what makes you think a new lower is going to solve the problem? Unless you walk in with your upper and a set of pins and keep trying lowers until you solve your "problem" or quit. The JP micro pin set seems the best place to start if it bothers you that much.

Legitimately, there is an armorers gauge for the ears for the front pivot pin on the lower to make sure it hasn't worn oversize. I doubt this is your problem. t's a parallel block 2 in x 0.750 in x 0.750 in. An end is machined 0.750 long and 0.515 in wide (+ zero/-0.002 in.) and shouldn't freely drop until the end of the gauge is level with the bottom of the ears.

Cyphertext has a point also, the sights are attached to the upper, so the bullet is going where the sights are. Will the wobble affect match grade accuracy? Probably, but then you didn't buy a Sport II to shoot itty-bitty, teeny-weeny groups.