Once you reach the teleporter shrine or get behind a couple of the gates and find the levers to start opening them, then you will not need to go up the ladders again. If you wish you can still return (if required) to get the scrying stone, and just not look into it.
