You need to visit the quarry tower (a little way to the north west) to get the key to the entrance of the mine.