Yeah it's problem because if you attack him before taking out the rest of the temple goes hostile and the game hangs. Or doesn't actually hang but take so long to get everyone in position that it might as well have stopped.