You can also simply run into the castle; after the guards stop you, pause the game immediately after the conversation ends, and then click inside the castle again. They will not be fast enough to stop you a second time.
Do not enter the castle either way before you have entered by invitation. If you get in early, talk to the duke and get a quest to find a cat, that quest will be broken when you do enter by invitation, and you will not be able to proceed. The only recourse for this would be reloading a save from before you entered the castle the first time.
You are not suppose to be back in the castle, so the NPCs inside do not have a change of dialog.
Welcome to the forum.
