What you had described was, you were not even reaching the elevator. You are still in the bandits cell area, where you need to get c4 to blow open the rocks that block the way. The elevator to the goblins is further up more
There is a hidden room in this area, where it shouldnt be hard to locate if you listen to what the bandits you had free told you. You find c4 in the hidden room, if I remember correctly.
