I thought that you just had to be a high enough level but there is a key on a table in a cave on the middle [ right ] side of map just before or right after you meet Mouse [ the bird watching guard ].
YOu go in there and fight some skeletons and there is a key in there. I never was able to figure out exactly what that key openend but I did go back to that same place you mentioned and was able to go in.
I entered at 5th or 6th level...
