You'd be down 1-2 companions (Halsin/Minthara for sure and maybe Wyll still).

Edit: Come to think of it, Larian may make your party react if you take no action. You might need to convince Gale to stay