I used to dock my sailboat at nearby Stock Island, near Mile Marker 5. You'll notice that US 1 Overseas Highway is navigated by Mile Marker (MM), from MM 127 in Florida City to MM 0 in Key West.
5 hours isn't much time. If you want snorkeling, the closest decent spot to Key West is Western...