There are other influences beyond Charisma, so just try with different characters.

For example, my Drow has good Charisma, but clearly merchants in the grove do not like her.