In some campaigns you could have been considered an Oathbreaker simply for allying with chaotic evil monsters.
This one’s on you, bro.
If you want to play a morally gray character, maybe Oathbreaker is the applicable Paladin subclass but you might have been better off with another class entirely.