When you first get to the elevator at the "top level", you have to get on the elevator, then click on the lever which is outside the elevator (I thought that lever was just to call the elevator if, e.g. you went down, then used your stone to 'port back to your tower, then took the shrine to High Hall, and walked back down, and discovered the elevator was still at the bottom).
You'll go down a level, then there'll be another lever, but that one doesn't take you down again - you have to fight your way through the second level of the cave to retrieve a missing lever, which you have to install in an empty 'socket' by the elevator, to go down to the third level.
