There are 3 quests that can get you an invitation to the castle, but no time limits on when you need to go. You are at a high enough level you can go whenever you want. I did most of the quests in Rivertown and Verdistis and cleared the Rivertown sewers before going (clearing the sewers made things a little easier later).
When you completed the plague quest the healers moved from Aleroth on their own and Seth left. You do not have to do the army quests; only 1 of the 3 quests that give you an invitation is required.
Thieves are friendly, so will cause a reputation drop if attacked directly (Poison Cloud cast near them should kill them safely). They do not know you broke the prisoner out.
I think Goemoe is more concerned with whether or not you helped the 2 wounded soldiers in Aleroth than with your general reputation.
