Keep exploring and do whatever quests you have. To get into the battle tower you need to do the Candles in the Wind quest.
There are minor differences in abilities between servants, though they may not even be noticeable (supposedly one necromancer concentrates more on melee resistance and the other on magic, but practically this doesn't have a huge impact on your creature, with a couple people saying they didn't notice any change in stats or effectiveness). I didn't really like Igor's voice or Allan's personality...
The main difference between servants are the platform upgrade quests and rewards (see
Sentinel Island Servant picks....).