Assuming you are working on a mill or drill press.
I will also assume you have the barrel rigidly fixtured.
You cannot do this with a hand drill.
This is how I would tackle the problem...
You're going to need a couple more of those tiny drill bits.
Flip the workpiece over and drill from the other side. Go very slowly and use cutting fluid ie. Do-drill or Vipervenon cutting oil. This drill bit used here is sacrificial as it will get wrecked when it reaches the broken stub of the first drill, that's why you need more than one.
Once you have reached the broken bit from the other side, back out and clean out the hole best you can then take the broken shank from the first drill that broke and insert the shank in the hole and use that as a punch to try and gently tap out the broken stub.
Once you have the stub out, chase out the hole with another new drill bit. Again. go slow and use cutting oil.
Since you used a #54 drill instead of the chucking reamer, the hole will likely be slightly over size for the factory pin, so...
You should pull the new sight out of the barrel mortise and clean all the oil off with acetone. Install the clean & dry sight into the mortise then bump the pin in place. (note you need a "cup tip" punch to install the pin). Once it's all back together, apply some Loctite 290 and allow it to cure. 290 is wicking grade and will seep into all the voids around the sight and secure it even if the pin is slightly loose.
Without seeing or examining what you have there, that is my best advise. Good Luck.
Cheers
Bill