If you fight the Spectator with the drow the difficulty drastically changes.
You have a chance to fight him alone, but you have to get to the solution of the problem before the fight: it's kinda a reward for a smart thought
Out of interest, how are you supposed to fight without the drow? As soon as combat starts he starts to remove petrification
