I don't really believe in making the game more difficult just for the sake of it being more difficult.
What's the point in curating the player's experience so much instead of giving them a choice?
You have all the instruments to make the game more challenging for yourself - and, at the same time, casual players retain the option to make the gameplay more forgiving.
Everybody wins.