That's exactly what happens. It's very sad.
All I can suggest is, call it out when you see it and don't be part of the problem by doing it yourself.
There was a very interesting article in one of the (centrefire) shooting mags recently - if you count shooters as one unit i.e airgunners, shotgunners, rifleshooters, we outnumber golfers by a considerable margin. The perception seems to be that it's a rare and exotic pastime. We may be niche, but we aren't that niche.
But golfers wouldn't tolerate the shit we do. Golfers wouldn't let themselves be marginalised. Golfers aren't afraid to tell people about their hobby at dinner parties.
The community as a whole needs to buck its ideas up, or it will deserve everything it gets.