It's not a flat -1 modifier.

But yes, charisma is acting very weird. Low levels only remove bonusses of opponents, not add to your own roll, and only 4+ add to your own roll... that's what my own tests showed anyway.