The teleporter (Camp Sentinel View teleporter) you need is on the west side of the island in the middle of the channel, on a ledge directly above the left Nest Tower as you approach the cliff. This brings you to Camp Eagle's Nest, where you can access the bunker.
The teleporter by the blind philosopher will be activated once you disable the barrier.
There are another couple caves in the starting section (one accessed from a ledge on the cliff a little to the left of where the bunker's anti-dragon shields end, the other across from the Grand Knight shrine, near the source of the shorter waterfall), though you might need to level a bit first.
