If you don't want to look at maps - the short answer is there are TWO doors to the east.
The one to the north that you probably got to first cannot ever be opened from the side you are on - it can be opened from the other side - there is a lever.
There is a second door in Fergus' room - so obvious I've overlooked it and headed north with the key myself once or twice <blush>. That is the eastern door you need to use the key on.
