Drill and tap an M3 hole in the silencer adaptor - screw in an M3 screw and lock it off.
Or even just use a small jubilee clip to attach a bent bit of metal to the body of the silencer adaptor.
Depending on the diameter of your silencer you may need to make the front sight taller than the original and in that case you may need to make a longer rear sight blade to match.

I have done exactly this mod myself - works fine.