No, unfortunately there is no way to remove an avatar character from the party.