You don't have to fight them, you can just walk away.
You mean over wayshrines and never come back again?
Yeah, just walk away. You get to have your cake and eat it to. Just make sure you don't need to go shopping before leaving the island and you've reached the height of roleplay compromise.