The dolls are not selected by default when summoned. Now that you mention it, I don't think you can individually unselect a character in the group, just select the lead character again and Shift click to add specific followers.