The thing about elemental resists is that while they can reduce your damage dramatically, it can also go the other way for enemies with low resists to certain elements. This means always keeping several staves handy, which is definitely tough sometimes.
This is true, in theory.
But, in practice, how many enemies actually have elemental weaknesses? I noticed 2. Fire Slugs are weak to water (-50%), and some skeletons are weak to fire (-10%), but nearly every enemy I saw was either slightly resistant (10-20%) to some or all elements and possibly highly resistant (40%+) or immune to another. All undead are immune to or healed by Poison, Warriors constantly go Immune to Fire, there's a new Water-Immune skill several enemies use, and lots of guards have a flat 10-20% resistance to everything.
Staves (and especially wands) already have lower base damage than their physical counterparts, then they generally get reduced by resistances (or at least have to jump through hoops swapping weapons), and then their built-in statuses are blocked by armor too, and this assumes you'll have same-quality and different element weapons to even swap to, and that you take the effort of inspecting every enemy to see their resistances and swap weapons often.
It just doesn't seem very appealing to me. There's no reason not to just go Physical for better damage. Physical damage parties still have Crowd Control and utility without having to do any Magic Damage.
Sure, Larian could introduce some enemies with very high Physical armor, or very high dodge (this still affects Staff and Wand attacks) to try to make Magic users appealing, or force more-varied parties, but I still think the Magic drawbacks should just be addressed. Otherwise taking a token mage will feel more like a requirement than a fun choice.
