Most probably wind - as Flims points out, once you start to stretch the distance, even if there is no discernable wind at your firing point, there can be plenty closer to the target. Although it won't have the effect that wind at the muzzle has, it will still be enough to produce that kind of lateral drift. Try stringing up a hankie or similar material at the target - might help give you a read if there is no vegetation to assist you.
Good shooting!