I think that's gonna depend on player's preferences and if they are role-playing or not.
The only character I would say it's stupid not to get on your party is Lae'zel because she's the only one that MUST join you on the Nautiloid (SH is optional), so both of you have history already by the time you find her again, and curiously enough, when you reject her to join you, the game creates a journey about her (is like telling you, are you sure you don't want her) and to my knowledge, that doesn't happen to the any of the other characters.
If you save SH, it would be stupid not to join with her.
Astarion is the character that I find your Tav has to be stupid to let him join you but the rest are up to the player, I think.
