I'm about 45 hours in right now and My party is Lvl 4. It's been a lot of run and not TOO difficult up until today. Then Just following the various story threads, I came across the
and thought it was nigh on impossible (of course they were Lvl 5 so not totally surprised). I did end up beating them with a LOT of work and once I remembered I had that
special horn that called in three big friends
. But even then 2 of my party went down before I finished them all off. Then I started to go to a new area and it said I would find it "bitterly difficult" at my level. Okay, I realized there was some area down south I hadn't explored yet. Stumbled into the whole
and even though I did what I thought was some clever sneaking, exploring, etc. once I got to the
final part of her lair with the girl in the cage
once again I very quickly gave up. Now again, I'm only Lvl 4 and everything I'm encountering is Lvl 5 so I figure I've just hit it too early even the questlines led me there. There's still other questlines I could explore that will take me west, but I'm just trying to figure out why it hadn't seemed so difficult until today and suddenly it was like a nightmare. Just bad luck on part as far as where I decided to go and what I decided to check out?
BTW I spend a TON of time exploring and trying to think of solutions to puzzles and such so that's likely why I'm Lvl 4 at 45 hours in. Today just felt like a total grind the entire time I was playing. I thought I'd see if anyone had any insight. I'll keep exploring and see if I can find other things going on that don't feel impossible for me! Thanks!
You should be lvl 5 for that encounter, its doable at lvl 4, but definetly can be a bit hard then, depending on your experience with the game, lvl and party composition.
Im playing on tactician mode, only combat I couldnt win was a certain woman under a certain house in her lair at lvl 4, (her and her minions). Tried that 5 or 6 times (several Hold person spells at once towards party is no joke). Did it on normal difficulty first try after.
My point being, unless your playing in story mode, theres no shame in putting down the difficulty

.
That encounter was Deadly as hell until they added lvl 5 in Early access.
Personally I did it at lvl 5 with no hiccups and no Ogre aid in tactician mode.
Minor spoiler.
PS: If you dont want to wait until lvl 5, then there are some Ogres that can help you make it easy in blighted village, if you havent killed them that is :P.
EDIT: Read further on in your post, seems like you got them.
I played over 200 hrs of EA, but Im very thourough as well, Ive done all of topside act 1 (including mountain pass etc). And I have second part of Underdark to go. Currently at 55 hrs I think, so your not alone in act 1.
If your looking to lvl up to lvl 5 I recommend you go above the toll post (the safe route), quite a few mobs to get xp from. The sheep in swamp as well. There is also a quite hidden quest you can do (lockpick a certain chest inside the Grove, that will direct you to a place along water in the swamp. And offcourse the temple at the beach if you havent been. Goblin camp is also doable at lvl 4

.