Maybe in multiplayer co-op, some people felt the need to go to the teller of secrets every level just to make sure they're getting the advantage, while others felt that was a waste of time. Now obviously you could just a different game where people don't do that, but it's always annoying when many people use blatantly OP things like getting tons of additional stat points. Of course, there's tons of different things strangers and friends alike can do to annoy each other, but consistently playing with overpowered characters in an already easy game was probably especially annoying.
In single player, what other people think is a non-issue. But Larian has to balance the game both for multiplayer and singleplayer, and they tend to prefer to let multiplayer balance trickle down to the single player experience (they said something like that in an interview), and they must have felt the teller of secrets book refreshes harmed the multiplayer experience, so that just had to go in the single player as well since they don't want to balance them separately.
You could probably modify your character stats or add books with the editor, but I believe they changed it so books don't refresh no matter what in the vanilla game. Never understood the appeal of becoming completely OP myself, but I do get why min/maxing is fun. Your best bet is to figure out how to mod a little bit.
