I never left Fort Joy in my plays and never got more than level 3, so I cannot say much.
In my experience, if you do the ship, the two voidworms at the beach, ask the Red Prince (I always part after it but you get nearly 2000 xp) and fight the sceleton, you get to level 2 before entering Fort Joy.