If you move the rare mushroom in your inventory, or drop it, does your quest log icon under the character portraits turn red?

I reloaded my save at that part of the game, and even refusing to accept the quest he would ask if I changed my mind the next time we talked. I tried getting the mushroom before the quest, but that didn't cause any problems. If I talked to him before getting the quest, he always offered it, and afterwards he would ask if I had it (without the mushroom in my inventory he would ask if I found the mushroom yet, while with it in my inventory, he would ask if I found 'my mushroom' yet).

Are you sure you are talking to the 'mushroom farmer'? He disappears sometimes behind his house, and other imps can wander around there.

Do you have the patch installed?