Or, in my case, you could not notice the diary telling you to go back and talk to him, and exit from that level, which I believe you can't then get back to.
Which means I'm stuck carrying this thing for the rest of the game, 'cos it won't even let me throw it away. Fortunately, quest items tend not to weigh anything.