I was confused at that point because I thought High Hall was that big building behind champion's harbour (the one they won't let you inside) - it's actually a village northwest of the harbour (and as Candlebbq said, you have to walk the last part to it).
If I remember right, the third floor of the mines was down from the second floor, not up from the first. The up/down controls in the lift should be all you need.