You don't have to rush to get into Stormfist castle. I completed the majority of quests in Rivertown and Verdistis and cleared the Rivertown sewers before entering; some people also suggest exploring part of the Dark Forest first.