There are a few issues that stand out that contribute to this kind of failure:
1) Junior officers are promoted based on “hitting the wickets". They are evaluated based on how closely they follow a highly-scripted career path, not based on how well they performed. Junior officers are rated against each other, but the system is rigged so that those with seniority in grade will almost always be ranked higher than those junior in grade, even if the officer with less time in grade is exceptional and the officer with more time in grade can barely fog a mirror.
The end result is that as long as a junior officer doesn't massively screw up to the point of getting received of duty, doesn't fail a physical fitness test, doesn't get caught fraternizing, and doesn't get convicted of a crime like domestic violence, DUI, etc he or she will probably make at least (O-4) - a Lt. Commander and equal to a Major in the Army, USMC, or Air Force.
2) With this system in mind, it takes 9 to 11 years to get promoted to Lt. Commander, but about 80% of all officers who stay that long will make that rank. That's 9-11 years before merit really starts to come into play and 15-17 years before an exceptional officer has any real edge in promotion to O-5 (Commander) over his or her mediocre peers.
The problem of course is that the real talent in the Navy has often left by that point, while many of the rest are just hanging on to make the 20 year mark before leaving for something better.
Commander is by the way the usual rank of a destroyer or frigate captain.
3) The average surface warfare officer will have 18-24 months of initial training as an Ensign in various things navy before getting an initial 30 month sea tour as a Lieutenant JG. However naval vessels tend to be at sea only about 6 months out of every 18 months. There are work ups, etc, but the majority of that 18 month "sea tour" is spent tied to a pier.
After that initial sea tour they'll be shore based for more training and then as a Lieutenant, they'll get a second 24 month sea tour as a division officer, with the vast majority of that spent tied to the pier. Then it's back to shore duty.
A Lieutenant will get another 30 months of sea duty as a second division officer tour, and then go back for another 36 months of shore duty.
As a Lt. Commander, after a few years ashore the officer will get about 24 months of sea duty, usually as an executive officer, before being promoted to Commander.
So... at this point a Commander will have 15-17 years of service, with a total of 6-7 years of sea duty, but with not much more than a 1/3rd of that actually spent underway. At best it'll be in the neighborhood of 3 years actually under way.
4) That's not a lot of sea time by merchant marine standards. The merchant marine has minimum sea time and comprehensive test standards to be promoted, and one day of "sea time" is defined as 8 hours on watch. Thus, there is no time credited unless the officer is actually standing watch aboard ship. A second or even 3rd mate aboard a large vessel will have more sea time than the captain of a US Navy destroyer.
5) There's also differences in how things work aboard a naval vessel compared to a merchant vessel. The chain of command in the USN will have ratings doing the work, supervised by petty officers who report to the division officer who reports to the officer standing watch on the bridge. Which means the rating operating the radar says "hey chief, we have a ship bearing XXX at XXX yards". The chief reports this to the officer standing watch in the CIC, who then reports it to the bridge.
In contrast on a merchant vessel the officer on watch on the bridge is also the officer watching the radar, so the chain of command and the delay is a lot shorter.
----
In summary, you have junior officers aboard destroyers who have very limited time actual under way, who are not promoted on merit or ability, but rather on staying on a prescribed track and not screwing up to the point of being relieved, they are commanded by a captain who may have as little as 3 years experience underway, and they are using a command and communication structure aboard ship that is optimized for operating on combat operations as part of a larger fleet, under orders from other vessels.
The thing to wonder about is why there are not a lot more collisions.
