I snuck around him and sent just one person down for easier sneaking. If you completed the gnolls quest they recognize you and you skip the whole what's the password issue.

I just assumed he would blow me up. Didn't even try and talk to him.