Why? Dragon Age, Knights of the Old Republic, Mass Effect all had 3-4 party size.
I believe there is few important differences, you are missing ...
Mass Effect have 3 classes (Soldier, Technician, Biotic ... and yes, i know there are another 3, but they are basicaly just half/half hybrids) ... So, when you wish to have full acess to your party potential abilities, you need 3 persons ... everything abowe, would be just repeating of things you allready have. Also, it would cause problems since in this game you are giving them active orders, wich could proove dificiult with more followers.
Dragon Age have 3 classes aswell (Warior, Mage, Rogue ... and yes, again i know that they have 3 specialisations, but again when talking about interaction with the world it dont matter if your Mage is Fire, Frost, or Thunder type ... what matters is if you have Mage to interact with the world) Its best seen in Inquisition ... Warriors are smashing walls, Rogues are picking locks, and Mages have telekinesis they refuses to use in any battle scenario for some reason. ... I honestly dont remember 2 at all, and its fact that in Origins, it dont quite matter what party members you pick as long as you have at least one rogue to pick locks (and either mage for healing, or realy easy settings and lot of healing bandages). Still, once again ... adding another party member is unnecesary.
And Kotor ... well, we could say that there are Ranged / Meele / Force user ... but honestly, i believe it dont matter at all, who will you take with you. Maybe in their personal quests it do, but that is completely different story. :-/
In this game, you have a little more classes, and even if many of them are simmilar ... you can allways set them to stats ... You have tank (constitution), muscle (strength) stealther (dexterity), brain (intelligence), eyes and ears (wisdom) and finaly face (charisma) of the group ... so, in order to create "optimalized" party, you are suppose to have them all ... its ofcourse possible to focus on one aspect, instead of another, aka. specialize ... but once you have to scratch 1/3 of your options it, to put it simply, becomes frustrating.