You can avoid the fight yes. And no, I didn't have this issue. The issue I have with the fight though is the fact you can't loot the bodies and the chests after the combat ends. It gets back right into the cinematic.