You probably don't have the recoil spring seated properly. That's the only hangy-down part that might hit the take down lever when the slide goes back on.
If that wasn't the issue, and you are still having problems, try this. Works like a charm.
Before you put the slide back on...lever down...put an empty mag in the weapon. Slide the slide onto the frame and then all the way back. It will lock back. Flip the take-down lever up...remove the mag...my choice...then grab the slide and let it go home. That's it.
Dry fire and rack the slide a couple of times to make sure that it's working properly.
It's that simple...takes 15 seconds at the most.
Let me know if this helped. I'm always interested in learning new stuff.