It seems that you "should" teleport into the tower from the overhanging rock above the cliff. If you teleport straight up from the beach it seems to be buggy.