That is a little low, but still doable. In the next area you may have to avoid fighting and run for the exit, though there are health and mana shrines that refill themselves after awhile.
I completed most of the quests in Rivertown and Verdistis and cleared the Rivertown sewers before heading to the castle; some people suggest doing some of the Dark Forest, as well.
