Unfortunately, there is no way to remove an avatar character from the party.
Depending on how far you are into the game, if you are playing on PC and restart, there are at least mods to skip the first act (Joyless Beginnings), or just skip the beginning of Fort Joy (level 4; The Great Escape - Skip The Hold and Skim Fort Joy); I have not tried either.