ya, just continue into the castle. they get taken away anyway so it dosn't really matter until later in the game.