There is no rush to get into the castle. I completed the majority of quests in Rivertown and Verdistis first, and cleared the Rivertown sewers.
You can also get an invitation to get into the castle solving the Duke's murder quest.
BTW, the Frog statuettes lets you move fast, even when encumbered.
