This happened to me later on in a playthough - occured after the tiefling party and ended up doing the entire underdark without saving. Man, that was scary.

Reported as a bug.