There no rules where you need to do which quest 1st. The temple of doom is just another quest in the game. As long you defeat the bandit leader in the temple, you will get out of the temple in no time.