Look in your miscellaneous inventory tab; the imp gave you a crystal which you need to break (click on it) in order to return to the inn. Make sure you loot everywhere first (specifically the level-up jug), as there is no way to get back (by normal means, anyway).
