If your tower is under attack you won't be able to leave it normally until you've fended off the attackers.
Otherwise, try using the waypoint shrine in your master's quarters; it'll take you to any waypoint shrine you've previously been to.
Last time I went through my Leave Tower defaulted to the Battle Tower Shrine just outside. After that it'll change to wherever you happen to be when you use it to return to the tower.
