This quest can wait until after you have the battle tower (the top of the wall and quarry tower will remain accessible, as will the top of Lovis tower and a couple other places). In fact, it is rather difficult to get the key before doing the tower, given the number of high level opponents there. If you do get the key early, there is still no way to get to the mine entrance without becoming a dragon.
Welcome to the forum.
