There are other influences beyond Charisma, so just try with different characters.
For example, my Drow has good Charisma, but clearly merchants in the grove do not like her.
It depends on your charisma, and your "reputation" ... wich reflect how are you acting there ...
Whenever someone see you steal, probably fight, destroy things etc ... your reputation goes down.
You can check your reputation (no, its not how its called, but i dont remember the right word) when you examine vendor.