It's been a while since I've played this game (almost three years) so I'm having trouble remebering when and how to get my stones back from the imp. Do I have to find all of the council members before the imp appears in the Council of Seven building in order to get them back or is there another way?

I rushed into the Castle early on without doing a bunch of quests, so no I'm having to trek everywhere I go and it's annoying since I don't often have hug chunks of time to play the game (maybe about an hour to an hour and a half tops.)