When I entered Fort Joy I was usually just lvl 2. When I left Fort Joy towards the swamp I was usually lvl 5.
Doesn't matter at all of course if you are a genius at playing. For normal people like me I consider it extremely suicidal to go to the swamp with lvl 2.
At lvl 4 new strong skills appear at the vendors to buy/steal and an item with lvl 4/5 is much stronger than what you have at lvl 2.