Even later, when I asked Blake for the key (just to try him on) and told him about the Seekers for Truth he told me to bug out.
I believe there is a quest to get the key, received from the 3 people standing in front of the garden (may just be part of the main Nericon quest). Blake will not give you the key until after you have done one of the quests to get an invitation to the castle. I told him about the cult, and he just warned me to stay away from them.
