Are you sure you only have 4 characters who can become party members or is it that you can only have 4 at a time?
Not sure this has been revealed. Still can't decide if I like games that have a base camp where you change your party. Yes you get to have different conversations,
and have mission specific skills along, but I prefer to have my party, with warts and all and not have to take non interesting characters along just because the situation requires it, but that's just me.
Plus there seems to be a party size and additional companions.