I reloaded the first time I was summoned just because I was in the mood to explore the village, rather than hack and slash in the cursed abbey (though it is not that large). You get summoned the first time you try to leave the imp village, so it doesn't make a lot of difference when you do it.