You'll just have an unsolved quest and lack a bit XP. Though the game is more designed for good swords later on. The decision is up to you.
Kiya