The path up to the battle tower does branch, with one way leading to the bandit camp entrance and the other to a locked door and teleporter shrine. If you didn't run past any bandits, maybe there are just some on the other path.