You could always go for something that looks less "tactical" and more outdoorsy like a hunting vest, especially if you live near a more wooded area where nobody would blink an eye at a hunter's vest.
That being said, unless you're talking about a vest that looks like something S.W.A.T. or Military would wear with all sorts of pouches, straps, bandoleers and a plate carrier on it, then it's most likely that the only person who is going to think you look obvious is yourself.
Honestly, I see folks obsessing all the time over their attire, thinking that certain styles or even brands of clothing along are somehow tipping folks off that you're carrying a gun, if not outright telegraphing it to everyone in a five block radius.
Believe it or not, ordinary people don't go actively scouting everyone around them, attempting to determine whether or not they are armed. That's an overly self-conscious if not outright paranoid way of thinking.
Personally, I wear 5.11 Tactical Pants, one of the brands that so many folks insist might as well be transparent because apparently everyone, including folks who have no interest whatsoever in firearms or mountain climbing is aware of the brand and instantly recognizes it at a glance, then automatically knows that you're carrying a firearm.
Meanwhile, despite the fact that I've been wearing 5.11 Tactical Pants for over a year now, I've yet to have anyone eyeballing my pants suspiciously, much less confronting me about how my pants or the gun they have ascertained at a glance based on style/brand is tucked inside the waistband.
Realistically speaking, 5.11 Tactical Pants look no different than any other style of heavy duty cargo pants, and normal people don't survey everyone around them, taking note of their clothing, or attempting to figure out who might be carrying a concealed weapon based on said clothing.
Don't get me wrong, I'm sure that there are paranoid folks who do that sort of thing, but then again, they're probably the same sort of folks who believe in all sorts of conspiracies ranging from the world secretly being ruled by an advanced subterranean society of humanoid reptiles to the sadly common belief that the average inept, out-of-touch, anti-gun politician is some sort of evil genius who has formulated an elaborate, long-term plan to systematically rob all citizens of their civil rights and rule the nation by force, even though they cannot possibly achieve their goals within their own lifetime.
But again, people who aren't paranoid don't think that way, ergo the probability of the average person you pass on the street figuring that you're armed based on your attire within reason, is extremely low. Just don't go around dressed like you're part of a S.W.A.T. Team or an actively deployed Soldier and you'll be fine.
Honestly, the sort of folks who think that styles or brands of clothing are an infallible indication that someone is carrying a firearm is most likely into firearms themselves, and is probably the sort who also thinks that anyone who wears clothing with Military Logos on them is in the military themselves, or that anyone who wears a baseball cap must be a hardcore fan of baseball.