Sounds like a glitch. Try reloading a recent save, if available, or just leave the battle tower, maybe teleport back to Broken Valley and return. When you get back, take out the bandits (level somewhere else if they are too tough; if there are just too many at once, try withdrawing back downhill so they spread out a bit, and summon your creature to help).
There should have been a cutscene after you entered the battle tower, after which a cave-in (near the bottom of some stairs) blocks access to the elevator until you can clear it.