If you leave before talking to Vacca you can sell the gem for 5k gold (if it is in your inventory when you talk to him he will demand it back, and turn hostile if you refuse). It is not involved in a quest, or anything, later.