Totally agree. I think there's a good balance that could be struck between convenience and reactivity/immersion.
Maybe you could swap out your party quickly in a menu at camp and when you're done, then your party members can say their little comments. It would even give an opportunity for Companions to comment on being traded out for a specific other companion (I just imagine Shadowheart complaining that we're taking Lae'zel over her lol).
There are a looot of improvements I'd love to see with camp, but I figured this one was at least simple enough to call out.