Yes. You can toss a teleporter stone past the guards at the gate to get past them. Alternately, you can simply run in; after the guards stop you, pause the game immediately after the conversation ends, and then click inside the castle again. They will not be fast enough to stop you a second time.
WARNING: do not enter the castle this way before you have entered by invitation. If you get in early, talk to the duke and get a quest to find a cat, that quest will be broken when you do enter by invitation, and you will not be able to proceed. The only recourse for this would be reloading a save from before you entered the castle the first time.
