I would suggest that difficulty slider shouldn't be in linear correspondence how long will battle last or how many enemies will spawn.

So, for example, you could play challenging battles which wouldn't last long.