The way to handle that is to scale monsters up to the player's level, but not down. Thus, a group of bandits that were impossible earlier in your career eventually become doable, but never trivial. I hate trivial.