If you move the rare mushroom in your inventory, or drop it, does your quest log icon under the character portraits turn red?
I reloaded my save at that part of the game, and even refusing to accept the quest he would ask if I changed my mind the next time we talked. I tried getting the mushroom before the quest, but that didn't cause any problems. If I talked to him before getting the quest, he always offered it, and afterwards he would ask if I had it (without the mushroom in my inventory he would ask if I found the mushroom yet, while with it in my inventory, he would ask if I found '
my mushroom' yet).
Are you sure you are talking to the 'mushroom farmer'? He disappears sometimes behind his house, and other imps can wander around there.
Do you have the
patch installed?