The damage isn't random, I believe it is trap related. My party indicated that traps were nearby, but I could not see or interact with them.
I've reloaded the save multiple times and I'll get the perception roll to see my party found traps, and each time I can't see the traps and I can't interact with them. But I'll sure take damage for taking any steps further.
