That mushroom farmer can wander into the garden, and sometimes disappears behind the western edge of the house (so even the green cross on the mini-map is not visible). Try just waiting a few minutes by the garden entrance and see if he shows up (organize your inventory, talk to other imps that walk by, etc to kill time). AFAIK he shouldn't stray too far from the garden or area just south of the house.
imp village map - the mushroom farmer is at #14
