Personally I just assumed it's because this is Baldur's Sin: Original Gate 3 AKA they're still using the D:OS2 engine where they also had this limitation. Dagless seemed to imply there's a narrative reason for the party size though, unless I'm missreading him.
