The two cases where my oath of ancients paladin ending up breaking their oath were:
Near the cistern area of the lower city sewers, there is a group of Bhaalite assassins who I knew were going to attack me on sight. Sneak-attacking them with Astarion to initiate combat triggered the "You wanted a fight, well now you've got one" narration and counted as breaking my oath, even though the assassins are not neutral (being spotted results in an immediate attack).
The other case was more complicated, and I can see why it broke the oath but I still don't agree with it:
Also in Act 3 if you follow Astarion's companion quest to deal with Cazador, you end up with a choice of killing of releasing the vampire spawn that Cazador intended to sacrifice. The problem is, these are spawn that were turned and immediately imprisoned; they've never fed on anyone, and they're all victims. This makes the choice extremely complex morally.
I chose to spare them because they'd done nothing wrong, and Astarion's whole story is about how he can be better than his vampiric nature. Yet sparing them with the understanding that they need to keep their urge to feed under control, is considered an evil act, when it's far from that simple.
This is where the hard game mechanics run into the problem of having no actual DM; with a DM running the game they and the player can discuss the issue, and be more clear about the paladin's intent in what they're doing. A DM could also give more warning by threatening to change your alignment, or even just flat out say "this will break your vow".
I think short term the game needs to fix enemy groups that are misidentified as neutral when they're not, and it should really add dialogue tags to options that will immediately affect your vow, i.e- [Paladin] to the "correct" choice, or [Oathbreaker] for the "wrong" one.
