The path leading to the harbour splits, with the right branch going down to the harbour, and the left going up to High Hall. You can fly most of the way there, but the anti-dragon shield can not be lowered, so you will have to walk/run the rest of the way.