That's how I got out with my rogue in the first place. I jumped in the hole, then broke the wall of the gnome's cell and the combat started, opening all gates and my party getting "freed". Later after I resurrected them I tried jumping down with my whole party but didn't get the debuff.
I don't really want to go back to an old save because a lot of happened since and I like how stuff is developing story-wise.

Oof, that's really specific. Sorry I couldn't help!
Why didn't you just jump with everyone in the hole though?