I actually took a stopwatch and timed it, and there was a full two and a half minutes between ending my last character's turn and getting to do anything again, and that's with several enemies acting before that, and several pairs dashing together to use their turn.
Fights with this many enemies just don't work well in a turn-based game when I only have 4 characters I can control. Any of these huge fights would have been much better with 4-6 challenging enemies instead of a swarm of what are basically just fodder, in my opinion.
In the early 90s there where many strategy games with many units and they where fine to play. Allot of them where "you move all your units then the enemy moves all its units" and yeah, you could wait quite some time. Like the old Warsong/Langrisser and Daisenryaku games where especially like that.
That is why when games like Shining Force, Tactics Ogre came around, which added a system where every unit has a "speed" and thus gets its turn based on that, where more fun and engaging. The PS1 had many games that where like that, Final Fantasy Tactics or Vandal Hearts come to mind.
Baldurs Gate 3 uses a D4 +bonus for Initiative, not a D20 + Dexterity. They did that for the sake that the player can move more then one unit at a time. But to be honest, in many fights it would be allot better if you not move 4 characters at once and then can go on a restroom break. Being able to have 6 characters instead of 4 would also help the issue.
