There is a barrier blocking off the starting area of the fjords, allowing some exploration early, but preventing full access until you have the battle tower.
If you fly out over the water from the start of the fjords, you should see a bunker high in the mountains behind some anti-dragon shields and defenses. That is where you need to get to lower the barrier and disable the shields (there are a couple traders in there, as well, after they surrender).
You can either use a teleporter to get up there (the Camp Sentinel View teleporter, on the west side of the island in the middle of the channel, on a ledge directly above the left Nest Tower as you approach the cliff, will bring you to Camp Eagle's Nest, where you can access the bunker) or it is also possible to fly above the anti-dragon shield and drop down in human form, if you find the right spot.
