If you break the Oath you lose next to nothing - as stated above, solely Inquisitor's Might.
Taking up the Oathbreaker Oath gives you Spiteful Sufering instead.