also, many of the instances of charisma battles have different bonuses to different strategies. sometimes, intimidation gets a huge bonus (usually when trying to convince orcs), other times, charm works best.
frankly, I think charisma only affects how the character you speak to first reacts to you. if it is a positive reaction, they minimize their bonuses in charisma battles. I noticed once I hit 5 charisma, no opponent ever started a battle at better than (3) for their rolls.
I think the bonuses on YOUR side in those battles are due to the option you pick for the type of contest (intimidation vs charm etc).
I'm through hunters and into the dark forest at level 19, so have done a LOT of these battles. this is my impression of how the system works.