At this point it is obvious the game is geared around four players as its the default setting. With the exception of a couple encounters four has been adequate.
Personally, I think there is room for compromise. Make the default four, and then allow a Charisma bonus of +3 or +4 allow for an additional one to two followers. Will that make some people optimize for beastly parties with OP charisma classes? Sure, but twinks are going to twink regardless.