I'm gonna put this here because it applies to party size as well.
Encounters right now are all jacked up. A single Imp is 200 XP. If I faced an imp at level 1 with 1 other character in the party, I should earn 100 XP. Fight 3 and level up. So, the first fight alone should level you up.
But they've severely nerfed the imps, and they only give you like 25 XP or something for each imp fight. Intellect devourers are supposed to be 450 XP each. So if you fight 3 with just Shadowheart, which is utterly ridiculous because 1 is a serious challenge for 4-6 level 2 characters, you should be easily level 3 or even 4 by the end of the first encounter on the beach.
In short, the enemies you face actually should drive you very quickly to level 4, but they are so severely nerfed that it throws the whole system out of whack.
The whole game, however, works much better from a multiplayer perspective. If you have 4 PCs+1 or 2 (Lae'zel and/or Shadowheart), the encounters are more appropriate, and if you split XP, also more appropriate for a slower increase in levels.
The point, if they did this game right, the encounters are already balanced, with proper enemy stats, for party size 5-6 right from the very beginning. The imbalance comes from allowing any party size less than that, so they have to nerf every encounter to fit less than 5 or 6.