I like to explore, so I actually did find all the books and lore. (Gale got an inspiration point from it). Every monster in a video game is built up to be a big and scary.
But let's say I did exactly that. I read the in-game lore about the spider. And said, "too spooky!" So now what, should I come back in another level? Or three? Maybe the fight requires a special story item I have yet to learn about.
That is just your assumption based on ignoring the "stealth and exploration" part of my post.
The books are a hint. You can use that hint and stealth the caves and find out it's a giant spider, larger than the kind that surrounds it. You can examine it, which lets you know how many hitpoints, resistances, level etc. If you do not know how much damage on average your party can deal, and therefore how long it will take them to take monster down, any new enemy will be tough in the game.
This is an open map game. The devs have no way of telling what part of it players choose to explore first, depending on the plot. You were for example told to kill the goblins, yet decided to take a detour to kill spiders in a cave that clearly is not the goblin lair. And that spider cave is an entrance to an even tougher place. At which point the only way to make it easy for players who assume everything they find on their way should be easy would be to introduce level scaling.
And not every new player thinks like that, at least not in this genre (cRPG). The original BG games were like that as well; open maps where you could encounter monsters too tough to kill, and where the game gave you hints about them through storytelling.
