There is no reason not to have a 6 man party. If you want less then play with less Charakters or even solo. Adjust difficulty down and everything is fine.
But ruining fun of people who are used to 6 man party’s in dnd games just because it’s divinity3 styled is nonsense