I think he's talking about wandering into the fight between the rebel leader and the magisters and either not being strong enough to win or leaving. This kills the rebel leader and the sad woman at the camp won't remove your chains meaning yiu will only have one without (if you win the arena) making it very difficult to get off the island.
General rule of thumb, just because you can escape Fort Joy very quickly, doesn't mean you should. Try to be at least lol 4 and talk to the sad woman as soon as you can.
