Keep in mind that not only the bullet exits the barrel, but also gases. Lots of slow burning powder creates more gases than lesser amount of faster powder. These pressurized gases exit at roughly muzzle speed in average and add to the recoil, since they have mass too.