I personally prefer smaller parties (3 or 4), but I can't help but see that allowing a 6-people party is by far the better option.

Simply put, if you want to play with 4 in a 6-people maximum, you can always just leave 2 slots open. Ideally, the game rewards you with extra XP (many other RPGs already do that).

6-People Party Pros:
- More class builds/role combinations are viable
- More companion banter permutations and overall story

4-People Party Pros:
- Smaller/Faster combat encounters