There are 3 issues with climbing. First is that it makes it much more difficult to define the limits of a space, Invisible walls at the edge of a map become much more obvious. The second is that Larian built a lot of puzzles and secrets into the game. Solving these is supposed to give you a sense of accomplishment. With climbing a lot of that can be bypassed. Third it makes telling a story and scripting encounters much harder as players could bypass important parts with climbing.
Yes climbing is cool, and I would love to have it, but this game was not designed to have it as it would require major changes to map and story design.