This is a fairly early fight. Gameplay dictates it should be reasonably winnable. If you had to fight Kagha, a strong Olodan, two of her minions, several druids in the Grove AND more strong opponents, it might be a rage quit moment for many. The druids that fight on your side are pretty weak.
It's about as early as Flind, which if you opt not to Illithid her to fight for you is a tough fight against her, 3 ranged characters that after a few turns start multi-attacking for 3 attacks per round, a gnoll that shuts down characters bonus actions and a couple of hyenas.
While the 2 Zhents sit in their cave doing nothing.
Also, as early as the Ethel fight, which if you're unprepared can be difficult too when she splits into 6 all blasting your party with full power Ray of Sickness (She can TPK you right off the bat before you even have chance to remove some of her copies)
Not to mention, that the situation you describe of going against Olodan + Kagha together is one where you've picked all the dialogue options that lead to the most difficult fight where you neither joined with the Shadow Druids, nor convinced Kagha to go against them.
There are other dialogue fails in the game that will just outright kill you (Such as kissing the dying Mind Flayer in the Nautiloid wreckage... Goading Vlaakith to kill you... Killing a certain story important NPC...). I don't think we should shy away from a difficult fight, when that at least allows you a chance to redeem your poor decisions unlike insta-deaths.