I'm just saying, allowing a party of 6 would just simplify the entire balancing issue. Use normal stats, a party of 6, balance everything to that, and whether you play multiplayer or solo, the game is balanced right. Then you can go wherever you want and have as many in the party as you want up to 6, and the game would be either more challenging or less depending on where you go and what you choose to do. Steal a bunch of magic items from the tieflings to buff yourself up? Fights will be easier because you chose to steal better items. Play it honestly, and it might be a bit more challenging. Choose to travel with 4 instead of 6, the game will be harder. Rush to the Gith fight, and the game will be harder. Fight every minor battle first and level up higher and do all your little background quests and such, and the game will be easier. Then you can do away with all the Difficulty settings nonsense and just let the players play however they want.
And, again, you have to consider multiplayer with all this as well. If you play the game with 4 players, all 4 character slots are then full. There is no more room for origin characters. So, you never get scenes like Shadowheart at the school or the statue with her hand glowing all weirdly, or Astarion with the Gur, or anything like that. The only interactions you get with the origin characters in multiplayer is in the camp. I can't tell you how unrewarding that is. I've played the game several times through now with 3 Custom Characters and 1 origin character and even that is frustrating. I have to switch out the 4th party member whenever I'm about to trigger an event that would include one of the origin characters. Fight the windmill scene, switch to Wyll first by porting to camp, talking to my 4th member and dismissing them, talking to Wyll so he'll join and then going back to the windmill. Such a pain in the butt, and if you play the game with 4 players, that isn't even possible. At least with a party of 6, you can play with 4 players and have at least 2 origin characters in your party at all times. Then you don't have to play the switch origin characters game as much. AND, on top of that, if you are playing solo, you can have 1 custom character and ALL of the origin characters in your party so you don't have to switch them out at all. You can utilize ALL of their abilities and such in every battle and trigger all the cutscenes for each of them as you journey. SO much more fun and less annoying.
And why doesn't 4 players account for difficulty? Because you don't have room to breathe with a 4 player party. With 6, you can take all 6 with you for a more balanced and easier gameplay. Want harder, go 5. You've just increased the difficulty by 1/6th. It's not too severe, but it's definitely harder. Want even harder, go 4 party members. Harder still, go 3. With 4 party members, even taking 1 less is severe. There's a big difference in difficulty when you go from 4 to 3 party members and the game is balanced around 4. You've increased difficulty by 25%. That's huge! Drop only 2 members and you've increased difficulty by 50%. There's a huge difference in balance at that point.