Just to the south east of the castle entrance is a set of stairs up. South of the castle is a narrow mountain section running south; circle around the castle clock-wise until you hit this, then go straight north a few steps until you hit the stairs.