The Underdark isn't the friendliest of places so eventually you are going to need to camp. If Glut in in your party, you can't, which is annoying. Why can't you camp in the Ebonlake Grotto, you can camp in the Underdark fort and lots of other places.
Regardless, it is what it is and you say your farewells and he waits for you in the colony. ((Come to the Circle)).
You have the option to re-invite - only he doesn't join if you've change the party at all. Strangely, you can dismiss him, whereupon he waits in the Circle for you to invite him...
Or you can tell him the duegar are dead, and he wants to see for himself - but again won't join or follow you.
However, return the party to the previous setup and he does rejoin. It appears he binds to a character and not to the leader, and once bound, it's permanent. This doesn't sound right to me!