You can get to the camp from the side of the Mountain Pass, there is a ravine here, if you jump over then it can lead you to the upper outer levels of the temple, not far from the bugbears and the wall.
Oh yes, forgot about this, which was rubbish of me as that was exactly what I'd just tried and what brought on this wondering about whether there was any other way (if you mean from around Mountain Pass X:-141, Y: 470 that is).
I heard something about the drow being able to enter the camp at will? Haven't played as one yet.
Ah, I've not seen that either, but then both my drow so far were the kinds of people who would free Sazza to gain entry to the camp so her dialogue would have trumped any drow-specific lines. But going off the encounter in the blighted village, where a drow gets a pass from the leader of the hidden goblins, if a drow can get by then so can any other character with the Disguise Self spell who disguises themself as a drow. One to try in a future playthrough.
EDIT:
You can also featherfall+jump your way into the camp from the cliff where you find Arka's corpse.
Ah yes, so you can! I hadn't realised that was doable but have just checked it out.