I managed to bypass the issue by taking the road to Baldur's Gate without Astarion in my party. This time game didn't teleport me to the party camp and continued normally and next time I camped, Astarion said his piece again, but this time the exclamaton mark didn't stay on him
