Full party use is truly what the game should have been balanced around.
That would be difficult to do, because the number of companions can vary wildly. In one of my two campaigns, I am playing a feminist drow dark urge sorcerer and due to how things went, I only have three companions (about to face Ketheric Thorm in his tower).
What is needed though, is a very good explanation of party size limit. Perhaps more than four tadpoles close together will reach critical mass and cause a devestating explosion? Perhaps fast travel magic only allows a party of four? Perhaps Withers will get upset if you cross the four treshold? Perhaps there is an ancient prophecy that must be adhered to? It doesn't really matter which explanation is cooked up, but there must be one.