There should be enough, there's one for each trial but one of the trials is ruined. I forget what it was called, but from the entrance to the temple (heading towards the statue and the elevator) you need to go down the right-hand path which iirc requires a little jumping because it's partly ruined.
You may also be led towards the area by a displacer beast, which will end up in a room full of demons and some gnolls that will all attack you on sight (it's a tough fight unless you make ample use of area spells). On the lower level you'll find a horrible "bed" and there should be an adjoining ruined chamber with a spider corpse and the orb is just lying on the ground nearby (use alt/right trigger to see it easier).
If you find that fight too difficult, I believe it's possible to bypass the enemies with some jumping and stealth, maybe just send your rogue, but you'll probably want to use ample quick-saves as I can imagine it being really tricky.
I think this stage of the quest really ought to have quest markers or more direction to point you towards the remaining orbs; this is where in actual D&D your DM would be dropping increasingly pointed hints sending you towards the only path you didn't explore or the room you missed, or they'd just have the door open after only two orbs because you've spent enough time on it already.
