You can also just run into the castle. After you are stopped, click inside again immediately after the conversation ends; they will not be fast enough to stop you a second time.