So, I'd say that this is because the "Conversation" is, as a whole game entity, a conversation with Astarion. The other failing is that the way that the spell is written at the moment.... I'll quuote you the section from the spell feedback list:
"The other problem here is that its description in its spell card, its tooltip, and its status effect disagree with and contradict each other. One says that it gives advantage on all checks in dialogue (charmer against charmed), while the other says you have advantage on all charisma checks against the target. These are not the same thing... and it's concerning to think that Larian supposes they are. If the charmed tooltip is accurate (stating that we have advantage on all ability checks in dialogue with the target), we should have advantage on insight checks, as well as an Intelligence (history, religion, etc.,) checks we make in the course of conversation to mention, bring up, or respond to information... if the Spell description is correct (advantage on charisma checks), then we do not. They currently contradict."
As you illustrate her, it appears that the mechanical function is the former option - that we flat get advantage on every ability check we make while in conversation with the charmed person... which quite obviously is NOT how it should be. The annoying thing is that this has been noted for many months now, even before charm person had any effect at all in conversation... and they still haven't actually fixed it.
