Just run in. You will be stopped once, but if you click inside immediately after the conversation, they will not be fast enough to stop you again. Alternately, you can toss a teleporter stone into the castle and teleport past the outer guards.