I suggest trying the 40 a while longer. If you're like me, you'll start to appreciate the "bulldog" in that little pistol and you'll be grateful to have it if the situation ever arises when you will need it. Not that the 9mm isn't a perfectly adequate self-defense round, but the 40 is a notch better IMHO. You have the best of both worlds now, with 9mm for the range, and 40 for S/D, both on the same platform.
If you decide to part ways with the 40, though, and you want to let go of the 9mm barrel separately, let me know.
