I think what was meant by the party of 6 is that you can recruit 4 characters and reject other and play only with those 4, OR you could do a little switcheroo here and there to experience all 6 companions. But I might be wrong in my interpretation.