This system is well and truly the biggest flaw in the game right now and a massive impediment to its success. All the points have been made well enough for a long time, so I can only assume this hasn't been fixed for technical reasons.
It's a great shame, though.
Larian designed everything assuming that you will always have 4 players in the game. That's why the inventory is the way it is, too.
