Full party use is truly what the game should have been balanced around.
That would be difficult to do, because the number of companions can vary wildly. In one of my two campaigns, I am playing a feminist drow dark urge sorcerer and due to how things went, I only have three companions (about to face Ketheric Thorm in his tower).
What is needed though, is a very good explanation of party size limit. Perhaps more than four tadpoles close together will reach critical mass and cause a devestating explosion? Perhaps fast travel magic only allows a party of four? Perhaps Withers will get upset if you cross the four treshold? Perhaps there is an ancient prophecy that must be adhered to? It doesn't really matter which explanation is cooked up, but there must be one.
You are forgetting about hirelings. They are there for a reason. The party limit at a given time should be the amount of companion characters you could have had, +1 for when you play as Tav/Urge, and the player somehow notified that if their story has left them with less companions than that, they either accept the increased difficulty resulting from it, or use hirelings to replace the missing companions. if a Durge kills all 6 companions you can get in act 1, they should still be entering act 2 as a party of 7. Durge and 6 hirelings.
As for contriving a narrative reason to limit to 4, it had better be a VERY good reason, because if it's not leaving people in camp will still come across as narratively profoundly stupid.
