What book are you having an issue with? As I mentioned in your other topic, if a quest item book isn't given to someone, it needs to be read, possibly in a certain location or at a certain time.
There is a diary in Maxos temple that get put into the quest item category which is presumably a mistake, as there is no associated quest, and two similar diaries get put with the regular books.
You can not get rid of quest item books (etc) even after the associated quest is over, though often the quest item does get removed from your inventory.
Used keys are removed from your inventory. It is possible to Lockpick a chest (etc) before finding the associated key, though, so you can build up some useless keys in your inventory. Hopefully the next patch will add a check for that, so they either don't get put in the inventory or they get removed at some point.
Random books can be sold or destroyed (though I usually kept unique books even if they were not directly quest related). I started off selling them, but for only 3 gold each, I didn't have a problem destroying them if I was out somewhere and didn't want to head back to a merchant to clear out some room.
