You get XP from using your Charisma to win arguments.
XP is a limited resource in this game, so passing up on any XP opportunity is not min/max.
Therefore, if you aren't winning at least some of arguments, you are doing it wrong. (You really want to win all the ones where the enemy stays around after; you don't necessarily want to win the ones where they leave afterwards, because then you can't kill them anymore.)
So then you counter with, "I don't need to be good at Charisma, I can just savescum the arguments." It is true that you can. However, you can also be mediocre in combat and savescum the fights, so it's not a comparative advantage. When you can savescum everything, it's about building a well-rounded set of characters.
Social skills in this game are actually good. You are mistaken.
Charisma will hardly give you a substantial amount of bonus XP. And you can win enough of them with no points in charisma or basic +1 charisma items. Charisma is not going to be the difference between level 19 and 20 or 20 and 21.