Agreed. I am not demanding to be able turn difficulty back up again once I've chosen the easier path. I am now way past Fort Joy and every encounter I have at Reapers Coast is worse than the last. Enemy armour and damage scales up dramatically, sometimes with opponents literally one-shotting me, while my damage scales up very little by way of comparison. Every fight I engage I have to save even the most miniscule success and reload dozens of times resulting in fights that take 3-4 hours each if indeed I am able to beat my opponents at all. Its starting to drain the fun out of the game for me, seriously and I most certainly will not start over after 45+ hours.
