You can do either, if you missed it when you get the invitation, then you can use the teleporter stones to re-enter, it is in the armoury, the key to the armoury is in the study, which is opposite the armoury, it is on a shelf, unidentified, when you get it see Grischa.