Mister Monster's approach was very similar to what I did. First time I got into the temple, I went through that cracked wall (but climbed up from the west side).
When I went back, I had everyone sneaking before using the waypoint to the goblin camp.
The goblins near it didn't even notice me arrive and climb up the ladder.
I did have to fight the hobgoblins, etc sleeping near the cracked wall, but I was on a ledge above them and they couldn't get up.
