We've been there two or three times and always enjoyed ourselves. We stay at the Loew's Hotel which is a few blocks north of the docks but I believe that they had golf cart transportation to & from the docks. Had a great special seafood dinner at the Chart House restaurant (Annapolis Waterfront Seafood Restaurant | Dining with a View | Chart House ) which overlooks the water. We also called ahead for reservations and they had our names printed at the top of the menus. Slightly impressed my "Better Half".
We preferred to walk to/from the docks and enjoyed the people (the natives were always friendly) and the wife hit the shops while I hit every ice cream shop along the way.
I believe that the U.S. Naval Academy will allow you to walk around after you go through security at the gate. We both enjoyed looking at the stately homes and the academy chapel. Both are historic and impressive.
We both just enjoyed walking along the docks and looking at the boats, ships, etc.
I can almost guarantee that you'll enjoy yourself, no matter what you plan to do.