My problem with their implementation of Oathbreaker's has nothing to do with Rules as Written - it relates to game content that gets broken when you accidentally or purposefully break your oath.
You basically, In my experience, lose ALL access to Withers functions. You can't resurrect any NPC's or yourself, you can't hire mercenaries, etc. He states he doesn't know you and you need to resume your original oath before accessing him. This happens whether you are an Oathbreaker before or after you encounter Withers in the temple.
Which ever way you want to play an Oathbreaker or get stuck with it as a consequence of your actions, the game shouldn't break for you for taking a legitimate game choice as offered by the devs.
