My defense would be that "it IS a factory spring".
Seriously though, I've heard some people express the same concern about carrying handloads in a self-defense gun. Have you ever heard of a case when a private citizen involved in a self-defense shooting was taken to task for his trigger being too light, or for carrying handloads? I think as long as it was a clean shoot, trigger pull & ammo isn't gonna be an issue. if it's not a clean shoot, even a 20# trigger and factory ball ammo isn't gonna save your ***.