Head down to the basement first, to get the key for the door on the first floor (above the ground floor). You need to mindread the mute goblin for the information you need to continue. Once you know the demon's name you can get past the door to the summoning chamber on the third floor (finish of everything you can/wish to in Broken Valley and Sentinel Island before going past that point, though).

After mindreading the mute goblin
you can read the book he was thinking about in his bookshelf (you may need to scroll down to see/read the bottom of the book list), on the right as you enter the room at the far back, in order to remove a false wall...