Over near the wall is a teleporter that lands you under the fort in the cliff wall. Climb the ladder, slaughter the two below and you can either spare the two above and they well trade with you or you can opt to kill them as well. There are three levers up there that turn off almost all of the dragon defenses before the port. You should have no trouble getting to the last two after that.
