Don't go down Gavin's cave to Fort Joy, you can't get back up unless you have lockpick, but even then there is far better quests to level from. My suggestion is instead sneak past the creepy level 10 dude into the swamp and fight the skeletons then viscous voidlings near the ruins.
I like you, was struggling to find places to level up too. I provided it in my review.
