I'm not sure why I am having such a different experience... confused environmental effects aren't even that common in my experience, and you don't have to play with them at all.
Taking away environmental effects would make for an incredibly boring combat in my opinion. What's fun in always casting magic missiles or always casting Edritch Blast?

Environmental effects vary the very limited combat with action + bonus action, compared to the much more versatile AP-style combat in DOS.