This is a known issue. As a workaround, if you have 2 of your characters speak to the elves while the others deal with Peeper, that will keep them from joining (though you'll also be down 2 party members for the fight with the void chicks, if you don't take care of Peeper in the dialogue).