If you leave the escort quest too long, you can break it by doing one of the quests that will give you an invitation to the castle (the healers will move on their own). The abbey can be tough until you are in your late teens or early twenties for character level. Other than that, it doesn't matter which you do first.
