Well ... I am mostly fine with this. The party leader ought to be charismatic. Shouldnt it ?

At most I would complain that some games do this aspect better and if you have low Charisma, you cannot have much of a party to begin with.