Truth be told, I don't want to spoil everyone's story right away, so those I don't actively want in my party will most likely be not recruited/killed/sent on their merry way to begin with.
To me it's not about spoilers but sometimes your companions want to talk to you immediately after something happened...and this is obviously not doable if someone is back at camp. So you may be on to something...it might be for the best to only take three companions per playthrough to get the most out of their quests.
