I have done a fair amount of deer hunting over the last twenty years and one thing I have learned is that deer move a lot. Often, they move just as you squeeze off a shot. You want a cartridge that will be adequate even if the shot placement, angle of entry, etc. are less than ideal.
Besides, you will be hunting in thick cover. IE, there may be brush, small branches, whatever, between you and the deer. These obstacles can deflect even large heavy bullets, causing a different point of impact. What do you think would happen to lighter bullets?
Sure, deer have been killed with .22LR rounds. I doubt that caliber is legal in your state, but regardless, I believe it is unethical, outside a survival-type situation. And keep in mind that a typical 22LR has more energy at the muzzle than a typical .38 Special.
Paper targets or coffee cups don't move and are not (usually) in thick cover. Lessons learned from shooting such stationary targets, probably under ideal ambient conditions, are highly suspect when applied to hunting in the field.