As an engineer I took a materials science class or two.
It is true that the anodized surface is a form of color-enhanced oxidization and that it is harder than the metal underneath.

HOWEVER, removing that surface layer - which is only a couple of molecules thick - will not in and of itself weaken the piece of metal. If you sand or sand blast away a significant layer of metal, then sure, that will weaken the piece. But removing the oxidization layer will not in and of itself weaken the metal.

After you remove the oxide layer, it will immediately begin to oxidize again. That is because we live in an atmosphere that contains 21% oxygen. Oxidization is simply the process of the oxygen molecules combining with the aluminum molecules to form aluminum oxide. It is the aluminum version of rust (iron oxide).

Where oxidized aluminum is different from oxidized iron is that even after oxidization the aluminum molecules remain tightly bounded to the other aluminum molecules in the piece - so they stay put and form a barrier layer. With NORMAL iron rust, the oxidized iron molecules lose their molecular attraction (cohesion) to the other iron molecules when they bond with oxygen - and that allows them to and flake off - leaving fresh iron atoms exposed to the air - which then also oxidize and flake off. This will continue until the piece of iron is eroded away one molecule at a time.

As we have all read, bluing is a form of rust - or oxidization. Where it is different than normal rust is that it is done with chemicals in a solution - creating a controlled environment (instead of randomly by the atmosphere) and it bonds a layer of other molecules (from the chemicals used in the solution) to the surface of the iron to keep it from flaking off and exposing fresh un-oxidized iron underneath.

So, all of that is to say that if you can remove just the anodized or oxidized layer from the aluminum WITHOUT removing a significant amount of metal, you won't weaken it and a natural oxide layer will rapidly form restoring the hardened surface and corrosion resistance. I would think bead blasting would be about the best way to gently remove the anodizing while removing the bare minimum amount of aluminum.