I don't think they are. Aren't dialogues closer to scripting than a system? Systems suggest to me rules and verbs on account of players. There are no "ask", "question" "flirt" options. Just manually connected nodes with no systemic meaning by themselves, used to deliver mostly uninteractive content (text or cutscenes), and in the best case scenario feeding back and forth to game proper systems (reputations/character build)
They are systems.
You have the systems of dialogue selection interacting with responses and other associated effects (Reputation system interaction, event triggering, stat based interactions etc). Even when it comes to the selection of dialogue there are systems, such as having indications for different options having a certain tone to them (Such as indicating temperment like having an angry face denoting an aggressive option or a clown/jester face for a light hearted option)
Just because the default system isn't complex, doesn't mean it's not a system. The very base of it is the simple "You select a dialogue option > NPC reacts to dialogue option" which is still a system.
Being purely scripting would mean there wouldn't be dialogue options because there would be no system that incorporated them, no ui for dialogue options and no deterministic response to options (It'd all be just one predetermined script based on pre-existing parameters set before a conversation took place)
You can boil down combat into similar terms if you try hard enough. It's all just "Use Attack on Man" and then the opposing response of "Man uses attack on you" - The depth of the system comes in the form of the options presented, how the target reacts to those options, how options build off of statistics... Not unlike dialogues.
To really shake up conversations one would need to abstract conversations and adapt them into a gameplay system, but I am doubtful if it is an objective woth pursuing. Games struggle with far simpler concepts (like crime) and I doubt one can come up with consistant ruleset for "developing a relationship with a fellow sentient being".
It's not really THAT hard. It's just a matter of refining the "Reputation" system to be less rigid, or at the very least less 1 dimensional.
Most things can be done via a complex reputation system that better tracks your actual actions and dialogue options (Rather than being limited to a handful of specially designated dialogue options and MAYBE a quest resolution decision). Some games already do things like having most dialogue options have some sort of designation (Pillars for one... Most dialogue options represent one of the personality types) it's simply a matter of translating this into more varied responses and also including non-dialogue options into the same system (Which some games also track, Dishonored tracks lethality and will ramp up the chaos the more you kill. While games like Cyberpunk 2077 will track how you complete certain missions, lethally, non-lethally, undetected or guns blazing)
You can even use a dual axis reputation slider with something like "Respect/Disrespect" and "Like/Hate" allowing for more overall dispositions as opposed to a binary "Love/Hate". Or even more complex reputation sliders for yet more dispositions and overall interactivity.
The most complex thing would be the lack of consistency required. In the sense that, you can't just blanket a system across all characters in the game. But rather tune their individual personalities, things they like, things they dislike, things they'll let slide if they like you etc. Since typically games simply paint swaths of characters with the same reputation normally based on what "Faction" they belong to. With only companions getting more unique personalities (For example, Karlach likes you being nice to people. While Astarion dislikes when you're nice to people)
