I think you had the right answer for 30 years: don't use a pocket holster. I don't.
Just my opinion, of course.
I can think of a host of reasons why you should use a pocket holster for a pocket pistol and not one why you shouldn't - UNLESS your pocket holster is incorrect, of course.
The answer to how you should be removing your pocket pistol from your pocket is "carefully". There are so many small guns in pockets I can't give a better blanket answer but I do recommend two things:
1. Make sure that the holster doesn't come out of your pocket.
2. Practice. and more practice. Because when you need it you'll be fumbling for it in a hurry - get some muscle memory!
***GRJ***