As I said in another thread, you missed a bunch of stuff, which should be enough to get you to level 3. That's when it's reasonable to take on level 4 enemies and win.
Plus, once inside the fort there is about 700 gold from completely unguarded and easy to steal paintings in the fort, which should be enough to equip yourself with all the skills you want. Combine those two and you should be ready to go.
