You can also run into the castle (though you are probably done by now). You get stopped once, but if you click inside as soon as he is done talking, they are not quick enough to stop you again.