Well, I mainly meant the specifics of your own resources, game mechanics-wise. I don't care about the Battle Tower as an actual in-game location, but rather because it has your servants, which, by extension, are responsible for potion and enchantment recipes, skill upgrading (and unlearning), and your Creature's parts. And of course, your coffer, which is an item reserve.

Having your servants displaced to a different location, or even losing them altogether (assuming you don't lose recipes and creature parts in the process), isn't the concern, really.