I'm pretty sure getting the gem before the quest will not cause any problems. Head to the warrior's guild next (north of the elven village) if you wish (after doing one quest and getting the second to retrieve the dragonheart gem, you should just have to talk to the head of the guild again to complete that quest). You will need the gem for something else later, as well.
I don't think the speech option that you had killed others like him was intended to mean other dragon riders, just other badguys. Zandalor kills a dragon rider near the start of the game and you see one awhile after going to the castle, but I don't recall fighting any dragon riders before the one in the imp castle.
