Isn't the overriding goal to 'escape to lands controlled by the Black Ring'? That's a reason enough for other escaped Sourcerers to be traveling in the same direction.
True, but how many would still be in the area of Fort Joy? I would think most would have fled when the purging first started, if not when Bishop Alexander gained control of the Divine Order (since his anti-sourcery stance wouldn't have been a secret). I suppose there could be a safe house somewhere collecting sourcerers, who can only leave port when the weather is bad and the night dark enough a ship can sneak in and out of port without being searched, or something. I don't see how that would be any different than having a selection of companions in prison, though.