You really don't have to continue on with the main quest just to leave the tower. Just use the teleporter in your quarters. It's in the front room by the long table and you can get to the fjords entrance using that.