You agree to break the pact and forsake your father. That means the Gate loses its great general who contributed greatly to pursuing ideals that are embedded in Wyll's heart and mind.
I feel that the "great general" thing is actually something the writing fails at portraying. I found the game told me Ulder was a great leader, but I didn't feel I was ever really shown it. The Flaming Fist is shown as corrupt, and he spends most of the game as a damsel in distress. I presume he is portrayed as competent in his other appearances in the franchise, but he's important enough that we shouldn't have to dig for that info outside the game itself. It made the argument that he was vital for the good of the city fall really flat, at least for me.