If your patient, saving and reloading you can get 3, twice got 4 gems at every site. Red Ore mine, Patriarch Cave, Lost Tomb. This takes time and patience but if you want them, they are there to find. 2 from Lamotte, one from the sleepers if you want to waste XP, 2 from the merchant in Orobas Fjords, 2 from Beatrice in High Hall and then the ones you mine. Sould be able to get at least 24 or so. Used a bunch and had 7 left over, so you should be able to find plenty.
