What happened to everyone on your island? I always assume at least a couple people were smart enough to run to your tower for shelter. Unfortunately, it looks like only the retarded NPC that wants a book that's worth a skill point is left camping outside. It really doesn't leave anything to the imagination.
But why is that annoying? One personal reason... I always imagined that with your castle, in our case a battle tower of power, you get all the accoutrements. For instance the chef couple that was outside, Henry and Eleanor, are apparently dead. WTF??? That really through me for a loop. In FoV, your conversation with Ricky confirms that your character knows their fate. I mean seriously, you get a lavish throne room and all you have to eat is conjured dog food and scraps you've pilfered off other people like a beggar!? I really wished Larian removed that part of the conversation or at least added a third dialog option that states they are now in your employ. I like the fantasy that my character has private chefs.
I must admit this bothered me: partly because I also expected the chef and his wife would take up residence at the tower, since it seemed an explanation for their presence, but also because I felt the fates of those left on both the island and in the likes of Broken Valley village were addressed in a slightly dismissive manner. Which is perhaps a subjective thing... perhaps I'd remained optimistic that Damian's arrival was at a glacial enough speed that most of the people had managed to leg it before things got really unpleasant, but alas not.
Then again, a few of the characters from the village did have cameos of sorts, which leaves me wondering; and the comment about its inhabitants
did only mention the Black Boar as far as I recall...
Edit: and was I the only one whose entire library was swiped from my coffer when FoV started? I'd spent ages building that up and was quite miffed.