Loaded a earlier save.
Anywhere in Act I I'm good. I can come and go as I please.

AS SOON AS I STEP INTO ACT II, Enemy of Justice.
Both the tower, and the Inn hate me.