That's the thing, they'd obviously need to balance it for one party size or another. And even if 4 remained viable I don't like the idea of how big the encounters would need to get to square off against 6 PCs.