I make the ingots in the garage in the Fall with all the doors open because quite a bit of smoke is generated. I use 100% wheel weights. I cast the boolits in the basement in the Winter because that's the time to do it! When the weather is good we are outside shooting or fishing or just enjoying the outdoors. I have a year's suuply of loaded brass, so loading is a Winter task as well.
I've been doing it this way since 1975. Haven't gotten goofy yet, unless you ask my wife, but her opinion doesn't count.....LOL.
In all seriousness, I read a report published by Lee, with data generated by an independent consulting agency. They tested air quality and people contamination when casting clean lead ingots. It particularly pique my interest because I'm an Environmental Engineer and the test methods were quite the same as I use to test factories. Bottom line is that bullet casting should not cause any concern.