In a perfect world all your mental stats would factor into your characters perception of every interaction. But not everybody even agrees on what they all mean, especially Charisma, but Wisdom too.
Another problem for me in D&D with this route; whatever your intelligence, any intelligence check you make is going to range pretty wildly for someone to be reliably a stup. Even those Int 1 animals only get (-5) to a check. Of course at that level you don't understand language, so even that range wouldn't be a factor.
I think a more interesting way of doing this would be making your perspective on a conversation involve rolling Int, Wis, and Cha and having the lowest roll dictate what responses are available to you. The Dark Eye does something like this with skill rolls I think. And of course there's the modularity of mind system that Disco Elysium does that makes it so interesting.
