I agree! I was wondering why I kept hearing dice rolling while I was in post-combat conversation with the hag. It turns out, it was someone in my party making saving throws for their life while I argued with her. Now that I know that happens, I quickly switch to another character and help them if I hear that sound in diaglogue, but it breaks the immersion to have to do that.
I agree auto-stabilize would be helpful in those kinds of cases. If the combat is over (or semi-over) and conversation is auto-triggered as a result, party members should auto-stabilize (which could be interpreted as another party member helping them in the background).
