I think this is it - this right here may be the important difference in viewpoint. Not only yours and mine, but of those who argue for the lenient ruling and against it.
Classes were never 'just' the mechanics to me. Each feature with any amount of flavour adds to the character's story and makes me imagine the things about their past it reflects, how they came to posses it, what situations it could've helped them in. Even in classless systems a skill usually represents something the character spent a lot of their time developing and changing it on a whim is cheap. It's one of many reasons why I personally didn't enjoy the story in DOS2, I just couldn't take it seriously and I worry the same will happen here if the system moves more toward disconnecting its mechanics from its story.
Originally Posted by Silverstar
Personally I just assumed it's because this is Baldur's Sin: Original Gate 3 AKA they're still using the D:OS2 engine where they also had this limitation. Dagless seemed to imply there's a narrative reason for the party size though, unless I'm missreading him.
A party of 4 is not a Larian thing but a 5e one. 4 is the reference party size and the whole system is balanced around it. Adjustments can be made, of course, but this is the default recommendation.