Ya I had same problem. I ended up selling the gem to them. I was able to finish the quest with the enchanter anyway. I think I was even able to do upgrade by paying some gold to Sasson. Did you try using Dragon Stone immediately? I condsider that but never tried.
