I personally prefer smaller parties (3 or 4), but I can't help but see that allowing a 6-people party is by far the better option.
Simply put, if you want to play with 4 in a 6-people maximum, you can always just leave 2 slots open. Ideally, the game rewards you with extra XP (many other RPGs already do that).
6-People Party Pros: - More class builds/role combinations are viable - More companion banter permutations and overall story
4-People Party Pros: - Smaller/Faster combat encounters