The first time you leave the imp village, you get summoned by a necromancer. If you try to leave by entering the BF, this can take awhile. Try walking out of the village (say, to the north west) and see if that also causes a problem.