I am not sure but somebody mentioned 15 or 16. I got 4 from quest in High hills found 2 in a cave with books and 2 or 3 in grave robbers cave. You can also get 2 from Khan but it will cost you dearly.
Remember, you have to give 2 gems to philosopher in exchange for 2 skill points.
You’ll definitely not get back any resources from disenchantments and charms are permanent
