The mushroom farmer is in front of (not inside) his house. He's walking around so you'll have to look for him to get the rare mushroom quest.

Ralph