I strongly, STRONGLY disagree with having lore books give you direct stat bonuses. The information in the books should be reward enough for reading them. It'd be far better if these books actually provided some useful insight and knowledge that could be applicable in future situations.
Reading them could unlock certain dialogue options in future conversations, e.g. reading a note with instructions to do something signed by "T.L.", and you later encounter someone with the initials T.L., and an insight check could make that connection and unlock the hidden dialogue option.
I wouldnt say lore books, but specific MAGIC books like these mindflayer books that you dont have to read, but just having them in your possession gives you access to them. Having a book on your posession wouldnt give a bonus, its a book you'd have to sit down and bring out the passages and compare and etc, but these you just touch and you get, the whole history of goblins? instantly, in your mind.. that's something major and not just a lore book but a tangible magic item that would reasonably give a modifier to knowledge checks related to goblins if you were touching it in a way no book lore or otherwise ever could.
These aren't books, like yeah having all books have a potential to be relevant in conversations would be good too.