Shooting is about doing it exactly the same shot to shot. If you are producing one hole groups then your sights are off.
Once you have a tight group, wind it into the middle with the sights. POI will vary between bench rest and Off-hand, don't change anything in your technique, just move your sights. That's why they're adjustable.