If you have a cleric of Tyr in your party (which I had in one of my playthroughs) he will immediately confront the paladins and call them out on their BS.