You're supposed to talk to the altar. That's your main goal, anyway. If you have already reached the haste switch then the altar is just a short walk from there, behind the glowing orb. The door on the way is not locked. And that should be it.
As for the door in front of the imp spirit, it opens when the door before it is closed (so it's open when you just enter the place, actually). Just move a barrel/box off any switch in the outside room and it'll open.
