Canon doesn't explain these things, so yeah, it's my headcanon, but since there is no canonical official explanation to all this, then my headcanon is just as valid as yours. Maybe Tav warned other party members that if they see them with AA again, this means AA enslaved them. Perfectly reasonable explanation. Maybe Minsc, Jaheira, Wyll, who stayed in Baldur's Gate, were looking out for Tav, because they were warned by Tav to defend them against AA.
So? It's like to say: "See? Sometimes this killer doesn't kill people!"
Meanwhile my headcanon is he just left them be because man is too prideful to go crawling after them. :P
Yes it's almost like there's a scenario in which he lets the player go freely and it's relevant to him letting them go to Avernus or something. Crazy I know.
