for everyone else with the same problem: You can use "leave" after leaving the tower the first time. Then you will return to the point from where you switched to the tower.