I'm actually fine with the 4 person party but I could go for a 3 person party under one particular circumstance--if the two party members you bring actively engage in conversation cutscenes. And I don't mean the odd comment here and there, I mean they give their input and even debate with the other party member/player/npc. Something like that would be easier to pull off with a smaller party than a large one and I will always--always--take dialogue immersion over anything combat mechanics related. As it stands, usually one companion makes a comment at random so you'll likely miss what other characters may have said.
So, OP, I don't think you're crazy or trolling.
