Fort joy isn't really easy by any means. Griff or the Houndmaster will dump you if you arnt leveled enough. You actually have to seek out the turtles/frogs/crocodiles to train up.
You forgot about Orivand which is also murderous. In general, the difficulty of fighting in DoS2 depends mainly on how well you know the mechanics and whether you have chosen good skills.
For example, if you know how initiative works in this game (completely different to most other games) most of the fights become much easier.
