Encounters don't have to be rebalanced if you have properly shared XP. A party of 6 will be lower-level than a party of 4, and thus will have the same effective power. A party of one or two will be higher-level than a party of 4, and thus have the same effective power.
Except that's not how it works with DnD system. Action economy, ability to maintain concentration spells, etc aren't determined by level. It will be either too easy or too hard for a smaller party, because of how DnD levels work.
