The main problem with 4 person parties is that you might miss out on some of the interesting NPC's because you don't want to swap out characters which you have already invested in leveling up and equipment and story options.

So what do we do? Look to Narlen Darkwalk! Design NPC encounters where they do not necessarily fall under party control, but yet will participate in adventures with your character. Isn't that the ultimate single-player goal, to not need a "party" at all?