If I'm not mistaken, when you open the chest and take the gem, you should eiter kill the goblins or hand over the gem. If you want to return the gem to your enchanter, you will have to kill the goblins before you can hand it over.
does it make a difference?
