Reading the books does have an impact on conversations in the Arcane Tower as well.
I believe this is likely to be something they're working on.
This is true. And this is something Larian can expand upon, and I agree they are working on it. The hints are there. A lot of books reference things you encounter in the game. Even Raphael is referenced in a book near Kevlor.
