I have this issue, in big battles.
And it is immensely worse with enemies that do nothing during their turn. Like frightened tieflings.
Because you see, enemies who have something to do sometimes need a bit of thinking time, but then they do their thing. Those who do nothing during their turn though ? It's as if they know they're screwed, so they will use dirty tactics like playing the clock to make sure you hate all the time that separates you from being given the victory.