I think you only require that one battle for the nearby merchant to agree to remove your collar. I've done four play-throughs now and never been offered a second fight.

J'aime le fromage.