Perhaps best to put it back, though now it's kind of too late. The now deceased elven village robs you of a quest. Other NPC deaths result in that same problem. None of these quests are vital of course, so you're not truely missing anything, nothing important that is. If you still have a savegame that takes you back to the "unpulled" sword, you may want to consider just taking it, killing the ghost and putting it back.
