I agree that it would be good to have a quicker, more intuitive way to reconfigure our party, but I guess you've now worked out that you can speak to any of your current party members and ask them to leave (which will have them go to camp and wait if you're not already there) then speak to the party member you want to join you.
In terms of party size, my understanding is that four tends to be what D&D 5e campaigns are balanced around, and my expectation is that this will remain the default for BG3. I like this, but not everyone does.
You can find a
massive thread on these forums discussing the pros and cons of different party sizes.
And there's been some
more recent discussion in the course of talking about party composition.
I think there was also a mod available to increase party size, but as I don't use it I'm not sure whether it works with patch 9. It's also possible that Larian will make party size configurable for full release, but I don't believe that they've issued any statement of intent on that topic.