Its very unlikely that the scope is moving enough in the mounts for you to visually see the difference, unless your also finding big dents on the scope.
So its more likely that you are canting the rifle and that you are doing this in a slightly different way each time you use the rifle therefore noticing the difference.
if you use a spirit level on the back of the action so that you know when the rifle is level and then use a plumb line viewed through the scope whilst keeping the action level with the spirit level to get the vertical axis right this will sort you out.