I'd say swapping armor in particular shouldn't be possible at all, but hey, it's not exactly a common occurrence anyway. Can't really go making a big deal of this.
Party control in Baldur's Gate 3 is a complete mess that begs to be addressed. SAY NO TO THE TOILET CHAIN