Well, I avoided the camp until I had met/defeated these major baddies somewhere else. However NO rep-loss in killing soldiers if you want the XP.
Kiya

Just take care NOT to kill one of the green(friendly) soldiers outside = rep-loss.