I didn't find this encounter especially hard, at 4th level. I went there as the last thing I did on the surface world, so I had all magic gear available on the surface, but nothing from the Underdark, yet. This was the one time in my playthrough that I actually used a Potion of Speed, though in retrospect I didn't need it.
I know it's not actually helpful to say, "lol it was easy for me" though.
I had Lae'zel (battlemaster) with 22 AC from Defense style, her armor, shield, and Shield of Faith spell from the Sword of Justice (oh and I raised her Str and Dex both 1 at level 4, so that was one more point of AC). She was the only person who stayed on the ground.
My other party members were Gale, Shadowheart (with Warlock Initiate), and my own Thief Rogue.
First round, Gale casts Jump and leaps up to a position on the rocks. My rogue dashes and climbs up to the top of the bridge, does a ranged Sneak Attack, and then Hides. Shadowheart just dashes and climbs up to the top of the bridge. She might have also cast Hex that round, or maybe in round two. In any case, first round was all about positioning except for Lae'zel, who immediately Frightened the leader of the enemy group with Menacing Attack.
One of their guys comes up top with us, eventually pushed him off. Enemy leader is running away, Lae'zel Frightens another one of them, Gale is blasting from up on the rocks with impunity, Shadowheart is Eldritch Blasting her Hexed target and dropping Healing Words (with the item that also applies Blade Ward on a heal) as needed, my rogue drops a torch and dips, keeps ranged Sneak Attacking, keeps Hiding.
Once we get them down in HP, Gale is putting them to sleep, then we're finishing them off with big crits. The leader comes up to the top and gets in Shadowheart's grill, Gale sleeps her, Shadowheart obliterates her with a 55-damage Inflict Wounds + Hex auto-crit. In the third round, I think, my rogue switches to melee, getting behind their archer guy up top, uses a Potion of Speed to attack three times and then Shove him off to his death.
I don't remember all the exact details, play-by-play for every round, but honestly we pretty much wiped the floor with them. None of my characters were even downed.
