When it comes to the big astral tadpole that unlocks the extra illithid powers...
1. If you haven't used any tadpoles thus far, you can just say no. 2. If you have used the other regular tadpoles, you have to make a high DC check to say no and avoid using it.
(Correct me if I'm wrong, but I'm pretty sure that's the way it works.)
*
Say what you will, but I consider the black veins all over my face and body to be a pretty big downside.