Yes, kinetic energy matters a lot, but only when when velocity and projectile shape and mass are such to allow appropriate penetration.

Imagine you are walking along the sidewalk and you are hit by a loaded dump truck going one mile per hour. Even at one MPH a loaded truck has big kinetic energy, but you just bounce off and walk away. Why? Because the velocity and shape do not allow penetration.

On the other hand, neutrinos, which travel at nearly the speed of light but have nearly no mass, pass through your body many times every second. You didn't feel them, did you?

To be effective, bullets need kinetic energy with both significant velocity and significant mass, and they need an appropriate shape.

Momentum, on the other hand, is a red herring. The momentum imparted to the target is less than the momentum imparted to your body. Did that shot you fired knock you down? If not, then the momentum will not knock down the person receiving the shot.