There are 3 quests you can do which will give you an invitation to the castle. However, there is no need to rush to get in; the area after that can be tough at lower levels. I did the majority of quests in Rivertown and Verdistis and cleared the Rivertown sewers before heading to the castle.
