I would take a wire brush to that holster!

From you initial description I thought it was a rough out holster, not a smooth leather outer surface.

I’m thinking you need to snug up your belt.

If you want more stick the rubber sheet option sounds good.