This is not a sudden shift. Shortest version I can toss out.
.40 arrives and they deliver successfully. Every gunmaker joins in; every ammo maker too, tons of LE, competitors too. If you wanted to detail a nearly perfect new caliber debut, the .40 S&W is your case study.
Then the AWB and 10-round mag limit. This makes .40 even better. 9mm sized gun, all limited to 10, huge edge to .40cal.
Then bullet technology improves (how much is up for debate) but we move to an era where MORE professional trainers are making a name for themselves and more folks are paying for professional courses and FAR more folks are talking about these things and the trainers have their own voice on the internet where only printed gun magazines ruled the day in decades gone by. And all those trainers are singing the praises of the 9mm, especially after the sunset of the AWB in 2004 and 9mm carry guns go back to holding 15-18 rounds. Faster follow-up shots, less muzzle rise, front sight back on target. And 9mm ammo has gotten better (they say!)
And then the LE organizations are replacing all the duty guns from the previous generation... .40 gets dropped here, gets dropped there, in favor of 9mm again. And the private sector follows.
You aren't poorly armed with a good .40cal. Are you better armed with a .40 than with a 9mm? Well, that's the kind of debate that keeps gun forums active.
I've never been a huge fan of .40, but the two main reasons behind that are because it had a huge role in the downtown of 10mm some 30yrs ago and because .40 isn't the finest round ever to grace my load bench. But in the past 3 years, I've added three elite .40cals to my stable and while I have no desire to carry .40, the three guns I've added are
phenomenal. And two of those three were gotten for better prices being .40's than they would have been had they been chambered in 9mm.