Just to confirm, you will fail the escort quest if you complete one of the other quests that give you an invitation to the castle first.
I don't recall anyone in the forum discovering or mentioning that failing the escort quest will prevent Verlat from moving to Verdistis.
