The entrance can only be reached after you have the battle tower.
From the quarry tower, fly in the direction of the wall heading south east. You should pretty quickly come to a walled enemy encampment (easily visible from the tower) with a couple fires, a teleporter, stacks of crates, etc. At the back there is a rock ledge you can climb up, or jump over to from the top of the walls, with the mine door in the mountain, framed by 3 rocks.
