Can it be done? Yes.
Is it a good idea? No.
I had tried this years ago with a .41AE, due to the lack of light for caliber bullets in .41 cal. They did fire but the bullets were all over the place, even at close range. I didn't try cast bullets but I'd say they will probably lead the barrel real heavily.
As a side note, I had to run the mouths of the .41 brass into a .40 SW die to get them to hold the bullets properly. They opened back up on firing, but this would shorten the life of the brass if done on a regular basis.