I'm stuck with Volo in my camp even though I never spoke to him in the grove or the goblin camp (specifically so I didn't get him in my camp).
Halsin I spoke to during his rescue and afterwards in the grove but he invites himself to the camp - I certainly didn't invite him.
I'm only at the end of Act 1 so your post has me depressed with the thought of all those characters getting into my camp. I run a tight ship not a drop-in centre.
