There is another way to get into the garden, but it breaks a quest though.