Hello JRDS,
I think it's more to do with damaging the fruit trees by eating the buds rather than physically damaging the structure. I suppose them eating the buds would make the tree less prolific in it's growth cycle but as I'm no botanist I can't say.
Hope this helps
Regards