You do not need the dragon gem for the Silver armour, which gets upgraded to the (invisible when equipped) Larian armour in the easter egg area. In my case I found the dragon armour to be a little better, but some have had the random stats favour the silver/Larian armour. Also, some people prefer the Larian armour because it is invisible on the character (for example, mages would not look like they were wearing platemail, but would have that level of protection).