Perhaps indeed your remarks were general, but they did answer some of my questions so thank you! Much appreciated.

I now know that those three battles actually are comsidered hard battles (I assumed they weren;t and should be considered 'casual' as they are standard story-line quests),
It's also relieving to know that these are some of the harder/challenging battles in the game and that not every battle will be like this, I guess I just have been unlucky in picking my fights.
When I just got out of Fort Joy I was lvl 7 and the first encounter I had was the freaking lvl 12 scarecrow...

I think I'll look into some guides on how to get more gold and some better gear, I already have some of the better weapons like the ice dragon sword, Vard Layal bow and Lohar's hammer,
But I'll look for some more.

I might also have to do some party optimalisation and respeccing, I could use some tips on that.
Currently I run;

The Red Prince Geo/Pyro, mainly invested in Pyro, a point in huntsman for traps and one in hydro for restoration and teleport. Build feels solid, nukes like crazy with all sorts of AoE.

Ifan Ranger, most points in huntsman, some in ranged. Build feels solid, perhaps a bit frail, but high focus fire burst damage.

Beast Aero/hydro battlemage, a hybrid fighter with water and air spells and lots of points into warfare. Mostly used for close combat. Does not synnergise well with Geo/Pyro because of this guy usually being in melee and all enemies and surfaces being on fire, although with his high HP and armor it usually isn't a problem. Not sure if to respec.

Fane Necromancer/rogue, by far my worst character, does nothing but cast some necro spells as Infect or Mosquito swarm, has no AoE and terrible action economy, will definetely respec this to a ranged character and look into how to optimise a necromancer.

So for Fane I would like to know where to get some better necro books, and If I can best respec into ranged instead of scoundrel?
For Beast I am contemplating in swapping out some Warfare points into something that enhances his casting ability and let him wield a staff or wands. I don't use his warfare abilities that much.
I was thinking to swap his warfare for summoning to get some extra support perhaps, but I;m not sure if that would work.