Yeah, it results in a lot of wasted time across a playthrough, and for me it even discourages me from swapping out quartet members.