TO the charisma, I feel a morality check would be a nice addition. Depending on what you do, your charisma may fail even if its high depending on your morality ala Mass Effect paragon/renegade.
This way, we just can kill Fort Joy in its entirety without any side effect at all.
