Also, if you increase party favor, your party members won't leave when you attack the grove. I have had 2 successful attempts with this context. They even have dialogue on the topic.