To be fair, the art of the D&D5e Forgotten Realms is what I assume you call "cartoonish" (Just make a search online) so I do not think Larian is to fully blame here. It would be different if the setting is Eberron or Ravenloft, I suppose.
If you are making a game based in a franchise, It´s usual to follow the design guidelines of the franchise, like in Mordheim and W40k. Mostly because WOTC will want his brand to be recognizable.
Everything continues to look exactly like a D:OS game and nothing like a Forgotten Realms game.
Taking a look at the WoTC official material from D&D beyond and the monsters manual anyone can see that they made a pretty good depiction of the creatures
I sometimes wonder if people do know how D&D5E´s creatures, uniforms and the city of Baldur´s gate actually look like before making comparisons...
The problem is the color pallette that makes very unrealistic, unlike BG2. The demons are very cartoonish in BG3
I suppose It´s a matter of tastes. I kinda like it the comic vibe.
And as I said, the videogame versions of the creatures really look like the D&D5e counterparts.