I agree, plus uninviting/inviting them into the party is a little narratively jarring. E.g. telling Shadowheart "we should journey separately for now", only because I want to check XYZ's inventory if they have item X. It seems strange. We never stop journeying together with all of them, we never truly part or separate... it's really only about who you take with yourself for a quick walk or a mission for the time being. The uninvite/invite lines make it sound like you break up with them as a companion for a while, instead of going out for a quick mission without that person.
To be honest, this was something I liked about Dragon Age Inquisiton - I don't remember the inventory management, but I remember that every time you left base/camp, you got a screen to pick the companions to leave with. So in a sense there never was an 'active party', you were all in this together and you just picked who to walk out with on that day on the fly. It really made it feel like "today I'll venture out with you, Dorian, Bull, etc", but it didn't feel like you're "parting" with the others. Here, some days I don't not take Lae'zel because I feel like she'll resent me for "throwing her out", lol (even if only temporarily).
And yes, it'd be nice to see each companion's inventory at camp, regardless of whether they are in the active party. At one time I had to be searching for a pair of gloves forever through uninvites and invites, and besides it being tedious, narratively it felt like I was messing with my companions' minds, constantly breaking up and re-joining them w/o leaving camp

(Also, ever since watching Viva La Dirt League's Epic NPC Man, I feel bad for skipping their invite/uninvite reaction dialogue... but there's only so many times I can listen to the same thing :D)