It's a difficult one for sure,
I wouldn't be happy about someone shooting over my garden.
The OP has confronted the chap in question and if he doesn't stop he's left with no option.
I understand reporting it is fuel for the antis but enough is enough, it's dangerous
If it's to the point where the police need to confiscate the rifle it's probably the best outcome

Martin