Making your own tools can be very rewarding!

Watch out for any risk of a taper entering the hole in the roll pin or it will tend to open it up as you try & drift it out & jam up. Less likely with a brass drift but if you find a steel punch that hasn't got a squared of tip then it might happen.

Just support things as much as possible, & ensure there is no obstruction at either end of the pin/ hole & get things lined up. Could you modify a G clamp or similar to wind a piece of drill rod or hardened rod to drive the pin out without the need to clout it, might be a more controlled way of exerting pressure from one side & supporting the other. Might also reduce the chance of slipping out of allignment too....

Best of luck