Correct, you can not get to the graveyard until after the castle.