Did you use one of those black mirrors first, before confronting the Sallow Man? Donno if it helps, but maybe it further progresses the quest.