From my perspective, 4 would be an issue only if I liked more companions and couldn't take them with MC because of limited space. Fights are perfectly fine with the party of four.