I tried fighting the first set of skeletons in the swamp using a level 4 party who had just escaped Fort Joy.
They are seriously hard at that level. Could be I'm rubbish at fights but I almost always lose if there are more than 2 of them unless I'm at least one level higher.
You can ignore them and come back later (that is one option and what I do as otherwise I just die). If you complete everything in the fort (and kill all the magisters) before leaving you will be on level 5 (or 6 if you fight Dallis).
Alternatively you can avoid them and go south and pick up some more XP (and decent free gear) on the other side of the Island.
If you must fight them then netherswapping your mage or archer with the Pyromancer on the tower helps.
I'd suggest stealing not buying the €245 skill books if it fits with your approach. I figure if you are planning to kill someone for their crimes you may as well rob them first. I can see this may be a morally dubious idea though and there are some (OK only 3) NPCs I can't bring myself to kill.
The skeleton just north of Lucian shrine sells decent armours as you know. There are also 3 places with rare level 5 stuff on the beaches just a bit further to the north of him/her (one of which will be on your map if you have been a good elf and eating things up) and there are various other places dotted around with high level gear to pick up.
