Do we really need an explanation for the party size ?
Is that something that bothered you in other party based RPG and/or is that something you think about when you're playing a DnD campaign ?
Not saying it's a bad or a good idea, I just wonder about the relevance/interest of it.
It did for me.
I always thought Pathfinder: Kingmaker kinda missed a golden opportunity here. If people in your party couldn't do their duties as advisors at the same time they were off adventuring with you, it would be a perfect excuse to limit party size by requiring a number of advisors running things at the capital exactly equal to the number of companions you had to leave behind.