You will be able to complete all your current quests after entering the castle, it is just a question of being at a high enough level not to have a problem with the area after that. I would say clear the Rivertown sewers, then go whenever you feel like going. I did the majority of Rivertown/Verdistis quests and cleared the Rivertown sewer before going to the castle, and had no problems after that.
