You can not get to the mine entrance until you have the battle tower; it is high above the goblin village, with no way to jump or climb there.
I got the key before the battle tower, and managed to defeat the first group of 4 black ring archers that come upstairs. I couldn't defeat the 3 black ring warriors on the lower floor, though, so had to exit out one of the side doors. I didn't notice the quest listed as failed until after I had the battle tower, but I think it said the guildmaster was dead due to Damian's poison.