This is a computer game... not a table top one. An active party of four is plenty for a number of reasons of which combat is (my subjective opinion) the main reason. Alowances need to be made for the new medium. Besides, you can have all of them in the camp (Early Access game).
Larian created meaty characters, well written (imo). They've shown their personalities and some their secrets . All are difficult nuts to crack. Two more in the party won't add anything except slow down combat even more. Besides, I'm starting to hear banter between them ( recent patches added the banter?). Plus, one of the characters I swapped in is hitting on the girls. Amusing... my dialogue options need to be updated stat.. (lol).