Why would you romance Karlach if you killed the Refugee.
I propose the romance automatically borked if you side with the Goblin.
Karlach won't stay in your party, if you side with the goblins, so there is no romance in that case.
But Dammon and the other refugees can die without it being really your fault in act 2. It really has to do with a bit of luck in a certain fight.