"If I just change out the barrel for a 38 special and keep the cylinder would it shoot better? Or would it make much difference or should I change out the cylinder too?"

It's a somewhat unsettled question as to whether S&W used the same bore diameter barrels for both the British and US Victory .38 models.

In any event, my experience has been that .357-.358" bullets seem to work OK in .38 S&W barrels. And if you use hollow-base wadcutter bullets, those will expand to fill any .38 S&W bore, just like a Minie ball. Unless the bore is rusted out, etc., I would not replace it. Ditto for the cylinder. Even though bored out, a .38 S&W chamber is not unsafe to use with .38 Special cartridges. The fired cases may bulge due to the slight diameter difference, and cases may or may not have a tendency to stick. You have to shoot them to find out. You can also always fire .38 S&W ammunition in a bored-out chamber if you choose.