Wait, I've got an idea - that wizard who gave you a sword, remember? The one Wolfy broke - remember? something crawled out - what if the wizard had discovered, your wife had escaped - she must be silenced. And in case, she had told you something: The thing in the sword he gave you, made you forget = your nightmares were a way for this knowledge to get through to you. And seeing these castle walls brings back a short memory bit about somethingg your wife had told you...
Just an idea, feel free to use it.
Kiya