Lae will attack you if you don't help her with the tieflings, after which she won't be part of the party and will probably be dead. I've never tried running away from that fight though, so maybe she changes her mind at some point.
You can side with the monster hunter and kill Astarian with him and he won't come back to camp.
You can kill the grove and Halsin won't join and Wyll with leave the camp.
You can eat Gales arm and he won't join.
Apparently SH goes insane if you don't let her join as per the video above
RP wise it ultimately isn't 'Your Camp" it is a 'Party Camp' and the others may want to keep each other around despite your wishes. Gale and Astarian both like SH so unless she is dead, neither one of them is going to kick her out. Wyll isn't going to want to turn anyone away because he sees himself as a hero.
Withers himself is working for someone who sees you all as strands of fate, so I doubt he can be talked out of keeping tabs on them regardless of you wanting them around.
