Yeah, I don't quaff speed potions until I see there's three rounds left. Don't get me started on Lethargic. Lethargic is the most annoying rule bar none, *especially* with a haste spell. What's the point of having that spell if it bars you from casting anything othwise useful. 'Well, otherwise Haste'd be too powerful!' - As opposed to what, Hunger of Hadar? That is literally BG3's answer to BG2's Insect Plague.
I get that these are 5E's rules, and not BG's, but shit, it's annoying. That needs a house rule.