Try dropping the mushroom, picking it up again and then talking to the imp again.
I had the same problem. Apparently it has something to do with having the mushroom already before you get the summoning quest.

I used to be snow white, but I drifted... Check out my Beyond Divinity fansite