I did not enter the castle using the teleporters, I waited until I got an invitation. I entered the kitchen from the hallway to the north. I spent a lot of time completely clearing out all the "fog" from the southern half of the map on both sides of the river and on the west side of the river I've cleared it out up to the Cursed Abbey before going to the castle. Could that have something to do with it? Maybe I messed up a quest I haven't been given yet?
