I hadn't at first but even after I finished the dishes I walked around the castle in and out around for like an hour, but it did not work for me frown. I saw some previous person said they had to go clockwise and not counter clockwise in the hallway for it to work, so I even tried that a few times but I gave up because I missed playing and started a new save. Thanks for the reply!

Since I entered the castle too early with my teleport cubes and talked to Janus before the invitation, I guess I messed it up.