That was also a possible thing in DA:O - I recall that if your relationship wasn't at a high enough point with Zevran when the Crows came calling he'd leave you to join them. I don't see an issue with that being implemented in BG3 - hell, DA:O didn't even really remove them from the party loadout, just greyed them out. And you were always able to get the gear back!... by killing Zevran.
