Safety is number 1, but be sure you are addressing the real threats to your health.
As my doctor told me when I was 60: "you're too old to have to worry about lead," so I didn't get a blood lead test.
No, never wore gloves. As of 2006, my blood lead level was STILL 6 (whatever units they use--maybe micrograms per deciliter or something like that).
You are infinitely more likely to die while driving, flying, or walking than to get lead poisoning. To be safe, wash your hands after reloading, avoid eating, drinking, smoking, rubbing your eyes, or picking your nose.
If you want to wear gloves, then be sure to wear a dust mask at all times, as you have more worries about dust than lead on your fingers.