There is a trader upstairs. If you speak with the ghost in his room, it reveals that the trader is a lone wolf and killed the real trader. If you confront him with this you need to fight him. He drops a letter and some sort of symbol. If you show the symbol to the front guard you can enter the sawmill without fight.
I think to free the elf you still need to fight them, but it puts you in a better position, if you can prepare the area. Additional you can also do a couple of quests involving the ghosts hanging out there, that can only be finished if the lone wolves are still alive.
