I also never used the charm spell. I played a good guy in my first playthrough and using such spells on normal people is certainly not good.
As for the Guardian, up to the end of Act 2 I didn't really have much against him, but now at the end I really dislike him.
What was strange in the first place was his reactivity if you didn't consume tadpoles. He stopped talking about it but still always mentioned when there was a tadpole nearby. I found it a bit sad that there was not much more to refusing all of the tadpoles.
The initial reveal in Act 3 wasn't bad when I experienced it. For me it made sense that I fought against the Githyanki as I didn't have Lae'Zel in my party anyway. Though I have read that you don't have a real choice at this point. That leads me to my main problem with him: You are entirely at his mercy. If I understood correctly you can only use the Orphic Hammer in the end, right? So you have to put up with him until the point of no return. And I found this a bit disappointing as he is far far from an unproblematic character as many have already enstablished in detail.
I felt that it took away Player agency where it would have been interesting and could be expected. There isn't a meaningful way to detach yourself from him. A moment that upset me for example was the encounter with Ansur. I played a draconic bloodline Sorcerer and after Ansur was hyped up a bit by Wyll and Duke Ravengard, I was excited how my character could interact with him. I was even glad I spoiled myself and knew he was already dead because there was sadly no real choice or reactivity at all. I had one answer because of the race that didn't change anything, my class didn't matter and the Guardian somehow killed an at least Adult Dragon on his own. And after the short dialogue where I couldn't even really critisize the Guardian, I'm forced into a fight my character would never want. After I didn't get the mechanic in my first try I just fled the fight, at least trying to forge my own childish headcanon how that encounter ended. And though my character now should know that Ansur was killed by Balduran the game pretended like the encounter didn't even happen. That was bad. I know that was a very character specific view of this encounter but for me it initially seemed like an important story beat that fell completely flat. Is there meaningful dialogue after you re-kill Ansur?
So even if your character starts to hate the Guardian, you can't leave. You have to put up with him.
With this in mind I just don't get the image that they seemingly try to paint with the Guardian. He doesn't seem good at heart or morally grey at all to me. Omeluum seems to have good intentions, the Guardian just says he does while killing the "Heart of the Gate" and manipulating left and right. In the end Mindflayers aren't the people they were anymore and I don't think I like that the game pretends they now more or less are.
To be honest, I get the feeling that they like their Mindflayer-Idea just a little too much. I found myselft thinking about the Source Collars in DOS II and comparing them to the Tadpoles. I thought getting rid of the Tadpoles at Moonrise and then feel responsible to put an end to it all could've also been an interesting solution. DnD has so many option to handle so many things, but at times it feels like the game brushes them aside so the story works (Gale's Scroll Of True Resurrection always comes to mind. Dying makes the tadpoles leave a body, so crushing the head and using such a scroll should do the job?).