I think the area with the most promise is the class system. The DOS games have had the most flexible approach, which actually I think has room for more definition. Although 5E might look like it has a hard class system, there are so many ways to customize it that it ends up being very open to player choice (multiclassing, subclasses, races, backgrounds). I would like to see a structured class system but with a high degree of customizability.
I feel kind of meh about classes.
There was some amount of appeal in DOS2's flexibility, with points providing both inherent stat effects, as well as access to skills. Allowing for builds to be made around various different aspects (Do you want certain stat effects? Do you want certain skills?)
I think the only major drawback to DOS2's system was how shallow it was. With certain stats just being optimal and there being very few skills overall (Which then also become more limited when you factor in the armour system it had) so often builds ended up feeling very same-y (Even more so with Divinity Unleashed mod whereby maxing out Aerothurge for the 50% skill cooldown was super important)
5e (And Larian's homebrew on it for BG3) has a lot of pigeonholeing involved in what otherwise should not be a cornerstone of character development. For example, races and backgrounds - I don't want to have to play a certain race or background just because it offers benefits to my build, I want to ROLEPLAY as a character of my choosing.
Multiclassing is cool and all, though 5e does have a bunch of limitations on it, such as accessing the mandatory Extra Attacks for martials, Feat selection, Spell Slots, having to manage casting stats... While DOS2's "Multiclassing" had none of these silly restrictions.
Meanwhile, there's also plenty of other ways to do character development. For example, Kingdom of Amalur had its standard skill tree that was populated enough where you could go pure Warrior/Mage/Rogue or you could make hybrids. Or you have things like FFX's Sphere Grid (Which is similar to Path of Exile's skill tree) which offers a lot of choice about how you build a character. You could also look at something like Grim Dawn's Devotion system, while it's only a supplementary progression system it could be tweaked into something fit for a full on character building format.
None of these require set "Classes" but will naturally flow into a class like build, while offering someone complete freedom if they desire.
The combat system is one that I feel concerned about. The armor system in DOS2 was a major issue for me, and I hope that nothing like that will be repeated. I think the overemphasis on surfaces in the past 3 games was excessive and I would like to see it reduced. BG3 casters also felt weaker than in previous games, which perhaps was deliberate, but it would be fun to have some of the old spell effects again.
Well... Combat systems are up in the air right now. There's some question about whether this new game is an ARPG or CRPG.
But yeah, hopefully that god(woken) awful armour system has been retired permanently.
As far as surfaces go... I think there's still a lot of potential in them. I think the best way to go about them is to provide a lot more complexity into what they can do, with a bunch more combinations and effects that can be created. BUT to move away from the whole "Literally EVERYTHING makes/interacts with a surface". That way they're an optional part of combat that one has to actively seek out and utilize, that can reward strategic usage and creative character builds. While people who aren't specifically messing with them will have to deal with them less often.
Honestly, magic in BG3 is absolutely busted. The only way it feels weak is when put alongside Larian's ultra-giga-busted Martial homebrew (Double standard action Haste, Tavern Brawler being cracked af, Elixirs of Strength out the wazoo, Fast Hands for extra bonus action...) but even then you've got the busted "Wet" abuse and damage riders per projectile...
DOS2 had by far the weakest magic, thanks to that atrocious armour system as it made you want to stack either physical or magical damage, which tended to end up with favouring physical because that's what your weapon users do (Also, enemies could be immune to certain magical elements further pushing you towards physical)
I can't comment too much on DOS1's magic as I didn't use it much, I quickly found the most effective strategy being just having everyone dual wield wands and regular attack everything...
In the past three games, DOS1 was also the game with the best attribute system. Although it was the least similar to D&D, it was a deep system with many interesting ways to play, and rewarded players for building attributes and talents around skills.
As far as attributes go... I think Pillars of Eternity has done them best. At least when it comes to character building (World interaction... Not so much)
Every attribute is useful for every build. It's simply up to the player to decide how much they value each aspect.
Where it went wrong was world interactions, that were based on the standard idea of each stat... Meaning my Mage was punching down walls because she prioritized Might (Increased Damage and Healing). While my Barbarian was making all the insightful comments and solving puzzles because he had high Intellect (Increased AoE sizes for his cleave effects and increased durations improving his Rage uptime).
Which is not particularly great for role playing... (Whilst also bringing up the idea that perhaps "Stats" and "Character ability" should be 2 separate systems. So that you can allocate your stats to best supplement your combat effectivness, whilst also choosing particular characteristics for your character)