So I started a new character. Now in Act 3 again. I killed the Ironhands so I had to fight the Steel Watch.
I killed the first Steel Watcher in Rivington. I also killed Gortash in the castle with all other enemies and SWs. The entire Wyrm's Rock is dead. The dragon, SWs, guards, everyone.

Then I move on to Lower City. It went fine. I went back to Wyrm's Rock. Fine.

So I'm not sure what the issue was in my previous game, but no issue from killing Gortash and SWs right after the ceremony. Granted it's the toughest fight. I don't need to go to Iron Throne or SW Foundry anymore.

I'm playing in Story Mode. I'm gonna continue LW, and fast travel back to Wyrm's Rock after a major fight.