From the quarry tower, fly in the direction of the wall heading south east. You should pretty quickly come to a walled enemy encampment (easily visible from the tower) with a couple fires, a teleporter, stacks of crates, etc. At the back there is a rock ledge you can climb up, or jump over to from the top of the walls, with the mine door in the mountain, framed by 3 rocks.
Unfortunately is sounds like you may have found where the door is suppose to be, but it is missing. Have you left Broken Valley and returned? If there was just a glitch loading the map, leaving and returning may get the door back, or perhaps exiting the game and restarting might help.
Try emailing support@larian.com; maybe they can restore the door, or place your character inside the mine, and you can just use the teleporter or dragon stone to get out when you are done. A similar situation, getting locked out of a required location, is described in the topic
Help...cant get to levers to lower force shields.