What reasons do the other prisoners have to head there? Why shouldn't they leave the player once they made it to safety and return to their loved ones, or go into hiding, or start their own Evil Empire in a galaxy far, far away?
Actually ... this are good starting situations for a really good SP - Campaign. Stabbey already wrote some good points, why not-sourcerer-companions wouldn't work, but think about the drama in the group itself.
You could deepen the concepts of the Obsidian Games: A bunch of misfits, who can't stand each other but have to work together. Only this time there isn't a glorious leader around, who keeps them inline, so the other members of the group are the only friends you have left in a world, where you only can count on your own kind and maybe not even them.
This could be awesome, if the writers give their best.
