This is actually possible when they say the equivalent of "Looks like your party is full" and boot you out of the conversation. Just walk up to the party member you want to dismiss, tell them to go back to camp, and then you can return to the one you're looking to recruit and have them join then and there.
