I believe charisma only gives you a "so-called" edge in the Rock/paper/scissors game. On my first playthrough I invested some points in CHA and had an 8-3 advantage, but after losing 3 duels straight (yes AI won 12 while I couldn't win 4, what a 50/50 game) I gave up this RPS thing.
Now I just skip the RPS and s/l. Much time saved.
You were just unlucky.
To get an accurate depiction of your chances in an 8v3 charisma encounter you would have to do at least a hundred samples, the more the better.
12 is nowhere near enough samples to deliver a verdict on the RPS system.
