I see lots of good arguments being made on various points here on this issue. So I won't get into too much of the stuff that has already been covered. With the current 4 party size, I haven't had much issue. Sure, sometimes larger battles can at times make me wish the enemy AI would move faster so I can do a turn again. Those battles, sometimes give a feeling that having a bigger pool of characters to act with would be nice, especially when it feels like one of the turns is being wasted picking up or healing a downed character over and over. Fights where the enemy spawns more such as the phase spider and mephits really have me feeling party size, but that again comes down to the turn ratio kind of issue. Ability to get any set of enemies' HP down hasn't been an issue. Times where my whole party manages to fail a perception check are annoying, but I am unsure if more party would help much in that area.
I'm on the fence right now with this debate on party size, perhaps when there are more companions revealed it will be easier to feel out the need for more slots. It would open up a luxury of opening a little bit more battle role versatility for clerics/shadowheart if they didn't as often have to be the dedicated healer. That said, as much as I like the notion of having more people in my party to be able to explore the respective characters banter,clashes and class abilities in one playthrough, I have to recognize that multiple playthroughs are encouraged by a restriction in that area.