I have come to realize the reason it is this way is because they plan on only letting you keep the party members that are in your party. Currently there is a huge topic to try and change this, so we'll see what happens.