Would it be a bad idea to keep only three party members at a time? If you meet a potential fourth member, then either send them away (not to camp) or swap them for one of the others and send this other member away. No other members in the camp.
I look at this from the roleplaying side. I can only have three members actively participating. I also don't want to constantly change the active members. I think that is unrealistic. And surely it's not fun for any of the other members to always sit on the bench in the camp and never be allowed to go into action. Why not just send them away?
But how would the game react to this? Will I need one of the characters later on and he/she will be gone?
When you play through the game again, you can put together a whole new party.