You can get into the tower on Sentinel Island after you do something at the windy peninsula (where there is a tombstone and three candles). Until then, there is a barrier across the path that leads into the tower, near the appropriately named Barrier Waypoint Shrine, west of the shipwreck camp area.
You need to complete everything that you want to do in Broken Valley and on Sentinel Island before taking over the tower (there is a warning in-game before the final section). At that point the 4 servants will move into the tower, and many of the NPCs and quests in the first 2 areas will no longer be available. There is quest in each map that will need to wait until later:
the currently locked quarry tower in Broken Valley, and the treasure in the ship in the tree on Sentinel Island that you need to be able to fly to get to.
Speaking of the windy peninsula, have you done any swimming?
Jump into the water to either side of that peninsula, and you can swim to the entrance of a hidden cave.
