I have played through to the end game I think on four different tavs now. Other than the illithid wisdom checks, I only used the tadpoles on my first playthrough and have avoided them since. It hasn't seemed to slow me down in any noticeable way. I just store them in my camp's traveler's chest and politely ignore them for the rest of the game.

Having played through both ways, it didn't really seem any more difficult or troublesome to make it through to the end. Others might disagree, of course. But I'm here to tell you I've foregone the diet of worms and had a splendid time doing so.

"Often forcing his victims to eat their own lips, he was caught and imprisoned for tax evasion." -Yellowbeard.