You assume that the barrel is "straight," whatever exactly you mean by that. You also assume that that means that all bullets will fly "straight."This may sound like a dumb question, but, why do most S&W revolvers have adjustable rear sights? The barrel should be straight; thus, any bullets should fly straight. Now, I can understand up/down. Different bullet weights and loads fall differently.
You have little to no evidence that either assumption is true. In fact, the existence of "windage"-adjustable sights hints that one or both may NOT be true. On top of that, different weight bullets seem to vary in horizontal POI, probably because of different recoil. This, btw, is not an assumption, but an observation.
AFAIK, the actual answer is that both of your assumptions noted above are false.