If you can't be seen from the road then shooting within 50 ft. of the road centre would be no problem as no-one could say they have been alarmed as they could not have seen you, so long as sudden noise does not come into it. I would imagine using a steel backstop within a few yards could cause alarm.
But even if you are firing directly away from the edge of your property where the road is so that safety is not an issue, it only needs a passer-by to see you with rifle or pistol to be frightened and voilà you have broken the law.
It has been said often enough on here as to how so many people are having a funny turn at seeing a four year old with a cap gun, so I don't suppose it is that far from possible that you could be reported.
The problem is that what most people would think unlikely to cause concern is not the point. It is whether the one person doing the reporting says they were alarmed. If they are anti-gun then they can easily say they were alarmed even when it sounds silly to anyone else!