Yea no. A random mysterious figure in your head just saying "hey, it's cool bro, use the powers, I'm totally not representing the tadpole btw" isn't remotely a good reason. If that's all they have, then it's as weak as I'm dreading.
I definitely think it's probably not a great idea to listen to the guardian but there is another interesting mechanic at work...the narrator. When you use the tadpole for a wisdom check the narrator mentions how the tadpole took something you will never get back. Now I don't think this is particularly likely, but how hilarious would be if the narrator was unreliable? Your character does have a lot of different voices in his head...what if the narrator is compromised in some capacity too? Probably not but a good thought experiment. I also think it would be quite the major plot twist if the guardian turned out to not be evil.
