Gotta get those EXP with smelly panties on.
And the Tooth rings.
Look, I believe getting the Charisma-related XP is important. I pump Charisma on at least one of my characters to make this happen (and to help with Bartering, since each point of Charisma and/or Reputation is worth about 1/8 of a point in Bartering). However,
I do not normally pump this with ability points, or at least not much -- I normally put one hard point in Charisma and leave it at that. That point, +2 from the Tooth Rings, +1 from Panties, and +1 from Considerate trait gets you to 5, which is enough to somewhat reliably win any Charisma challenge you come across without taking off a Bartering amulet (if absolutely necessary, break out the tooth amulet).
So I'm not saying to dump a lot of points into Charisma, or to get Politician or anything like that. I'm just saying not to neglect it in terms of itemization. And for what it's worth, if a spell ability like Aerotheurge also had jewelery pieces which could let you hit 5 with only one hard point in the skill, it would be pretty stupid to put a lot of hard points in Aerotheurge as well; it's the items, not the utility, which dictate the lack of hard points in Charisma.