It could be the stock mounting screws are flexing the action....if it's spot on outside the stock then the only variable has to be the stock. You do also change contact points when in a bullpup or std stock, but that shouldn't account for erratic shots.
I would look at the fixing points of the action and points of contact with the stock.... shouldn't be a difficult fix.