The hardest fight is admittedly the Githyanki fight. You are right, its brutal. They are level 5, and can murder you on the action economy. Even for experienced players it can be rough. You've got to really use the environment, speed potions, all the buffs, summoned creatures and anything else to win that.
I got exceptionally lucky on this fight as well, but they demolished me my first several tries. I think the thing that sucks about this particular fight is that it really feels like it's unwinnable (on the current patch with level caps etc) on your first go at it. If you walk up at level four and just have a conversation, you're very likely to get absolutely demolished by people who can take way more actions than your entire team. I only won because I knew ahead of time what the engagement was and prepped by splitting my party up. That said, it was still fun, I just have no idea how much will be easier with further patches or whatnot. Also, thanks for the tip on goblin camp. I still think isolating the warlord should be possible in some way, but it's good to know that there are other options there. I must not have explored thorougly enough. Oh and thanks for the inventory sort thing! I had no idea you could sort by weight.