I think I remember that I once read a post that if you get an invitation to the storm fist castle you can also access the garden.

There is always hope