Presumably they don't normally move into peoples houses do they?
And I'm guessing most people don't have too many young trees in their gardens either.
But even then they seem to populate many woods without detriment to where they live...