There are 3 issues with climbing. First is that it makes it much more difficult to define the limits of a space, Invisible walls at the edge of a map become much more obvious. The second is that Larian built a lot of puzzles and secrets into the game. Solving these is supposed to give you a sense of accomplishment. With climbing a lot of that can be bypassed. Third it makes telling a story and scripting encounters much harder as players could bypass important parts with climbing.
Yes climbing is cool, and I would love to have it, but this game was not designed to have it as it would require major changes to map and story design.
Excellent points. However, not everything needs to climb-able. Just place invisible ladders in key places that allow the Athletics skill to shine. Maybe give some visual clue that lets players know that a particular surface can be climbed. If it breaks a puzzle or bypasses something that should be difficult, then don't add a ladder there.
Realistically, the Persuasion skill should affect every conversation and every dialogue choice, but instead they pick key places to use the skill. Climb can be the same way.