I am okay with 3 and would not be upset with 4. To have 5 companions I think they would need for the game to be in hard mode all the time or up the general difficulty across the board.
People cryuing that "Story mode is too hard" goes to show that there's no making everyone happy in any case, when it comes to difficulty.
I'm already using the "Party limits be gone" mod, anyway, which luckily doesn't even lock achievements, so having a native and officially supported option would be nice in principle but make little difference at this point.
That said, holy shit if I'm not missing some option for "formation control" on the other hand.