I don't think that is true of the police, if you invite them in then you can ask them to leave at any time, & they cannot search, however if they have seen something illegal/suspect out in the open while inside then they can act on what they have seen.
(never invite a bailiff in as that gives them the right to re-enter at any time)
You simply need to ask what grounds they have to think xyz
driving unusually slowly at 2am, as above, points towards possible drink driving which is understandable grounds to stop you, but saying "I thought xyz" politely ask "why/on what grounds did they think that ?"
I find being polite & helpful works wonders and I speak as someone who has a neighbour making repeated false accusations against me, often to do with me shooting on my own land
it is their job to investigate when someone makes a complaint.