From the Fort there is a ledge around the side you can jump from to get down there. Trigger the Spectator with your most useless companion or a familiar, when the Drow are unpetrified, the Spectator will still attack them and thin them out a bit, unless this is unintended behaviour. The rest of your party can get as close as needed for ranged attacks, you may have to climb down the vines. Any Drow left that turn on the rest of your party should give you enough room to climb up the vines, run back to the Fort and hide around the corner. You can pop in and out of "cover" and finish them off, as long as you use the bonus action to hide they should not move closer.
The most direct route to this area is going through the Ruined Temple and doing that puzzle, this leads you into the Fort.