If you come in from the non-bandit side, just light the torches either side and it'll swing open. If you come in from the bandit side, there's a switch in the library room, near the back, that will open it.