I didn't hate losing those party members. On my very first playthrough, I had already decided on my party makeup based on personalities. And despite my decision, it was nice running into them later and still getting the opportunity to speak with them and see how things ended up working out. On subsequent playthroughs, I eventually cycled everyone into my party at least once to get their full story.
Also, the 4-person party cap doesn't bother me in the slightest. Smaller groups tend to feel more tight-knit and flesh themselves out better in the long run from what I've seen.
I don't want to fall to bits 'cos of excess existential thought.
