In addition to the army quests, there are two more that can get you an invitation to the castle: the duke's murder and the plague.