One side of the rivet will look slightly larger than the other side. Use the punch on the smaller side. I place the breech on a block of wood with a hole drilled in it.
Don't be shy, it takes a fair bit of whack to start the pin moving. Use a good punch as described above and preferably a reasonably large ball pein or lump hammer. Once the rivet starts to move it becomes easier so use less effort.
I believe the punch size you need is 4mm.
I use stainless roll pins to replace the rivet. Cheap to buy a pack of 10.