This has been requested a number of times during Early Access, and I agree that it would be great to be able to accept a new recruit and send an existing party member to camp rather than have the new recruit go to camp and have to go collect them.
As well as to be able to swap around party members more easily in camp and access everyone's inventories while there.
Personally I'm happy with a party size of four, though many people aren't. We have a massive long thread about this at
https://forums.larian.com/ubbthreads.php?ubb=showflat&Number=672266#Post672266.