The yellow followers were replaced by the current blue ones.
If your magazine is old enough to have a yellow follower it's probably old enough to be time for a new magazine spring.
Usually the early indications of a weakening magazine spring are failures of the slide to lock back on an empty magazine, followed by (or accompanied by) failures-to-feed involving the last round. Further weakening can often result in failures-to-feed earlier in the magazine load as the spring further weakens.
FWIW, S&W armorers are told to replace magazine springs in LE guns either every 5,000 rounds fired or else every 5 years (when magazines are left loaded). This is likely a conservative recommendation (better sooner than later when it comes to the reliable functioning of guns 'dedicated to defensive usage'). I've spoken to an armorer for an agency where their issued 4006's didn't start experiencing feeding issues related to weakened magazine springs until their folks had been carrying them fully loaded for 7 years. Better safe than sorry when it comes to something as easily replaced as a magazine spring, don't you think?
The newer 11-rd mag springs for the 4006/03 series have approx half of the spring painted light blue.
I've had a lot of our folks who carry 4006TSW's complain about how hard it is to load the 11th round into magazines, even after they've been carrying them for a year or so.
If you call S&W they might replace your older yellow follower(s) for free, and might even toss in some new mag springs (although mag & recoil springs are parts which are normally expected to wear and be replaced). Wouldn't hurt to ask. Probably wouldn't hurt to replace the old recoil spring while you're at it, either.
Considering the higher slide velocities often realized in the .40 S&W cartridge it's not exactly unexpected that weakening magazine springs might cause some functioning issues at some point.
Just my thoughts.
The yellow followers go back several years.