Specifically, a Third Model Perfected Single Shot pistol. In this case, the front sight pin.

I haven’t read anywhere that holes are tapered, so I imagine that the pin can be removed left to right or right to left. Or, are they tapered or is there a preferred direction of removal and replacement?

Further research suggests that holes are not tapered, but that Smith & Wesson installed pins right to left so removal should be from left to right and installation right to left. And so I was able to answer my own question.