You agree to break the pact and forsake your father. That means the Gate loses its great general who contributed greatly to pursuing ideals that are embedded in Wyll's heart and mind.
I feel that the "great general" thing is actually something the writing fails at portraying. I found the game told me Ulder was a great leader, but I didn't feel I was ever really shown it. The Flaming Fist is shown as corrupt, and he spends most of the game as a damsel in distress. I presume he is portrayed as competent in his other appearances in the franchise, but he's important enough that we shouldn't have to dig for that info outside the game itself. It made the argument that he was vital for the good of the city fall really flat, at least for me.
Ravenguard played a big role in Descend to Avernus, a tabletop campaign, that is basically the prelude to BG3. I guess, they were banking on people knowing his character.
"We are all stories in the end. Just make it a good one."