Probably the best way to deal with maintaining a challenge is just to have enough difficulty levels to satisfy everyone. I'm imagining Easy: combat is trivial, Normal: combat still pretty easy but not a pushover, Hard: combat is decently hard, but most non-idiotic builds will still prevail, Hardcore: Only really smart players and min-maxers will survive.
Yeah, I'm a big fan of difficulty settings and I think I've mentioned that difficulty settings provide a way to have our cake and eat it too.
I still don't really understand what theme you want talents to go under. Splitting social and combat talents COULD work and make balancing a little easier, but that means a lot more talents. There's only like 5 social talents right now. Keeping them in one list seems fine to me and makes for more meaningful decisions (do I sacrifice combat prowess for social skills, or vice versa?)
That's a fair point and it does add the dimension of choosing what you want your character to be. The problems arise when you take into account that D:OS was a very combat heavy game. Its difficulty was trivial, so your choices of talents didn't matter in a combat context, but what if it was more challenging and they did? You'd hardly have any choice because of the aforementioned combat-heaviness. If you could use your social skills to skip all/almost all fights except the most trivial of them that don't require any kind of combat talent investment, then great.
This over-reliance on combat is the true core of the issue. The difficulty was the only saving grace of the talents, because it was obvious that most of them were kind of thrown in there as last-minute ideas. The problem would be more pronounced if the combat was harder, like I said. Apart from splitting combat and social talents or NOT having any +dmg/+ap talents at all, and mostly avoiding combat talents in general, the other solution would be to make any playstyle viable to reach the end.
VTMB is a good example (at least the first 2/3rds) - it was balanced very well in the first 2/3rds of the game. Characters *focused* on any one of the main ways (combat, stealth, social) could progress. They ran out of time/money at the end and it's just a shooting gallery from then on, you were screwed if you neglected your combat skills. This way relies *heavily* on good writing though, and is much harder than endless combat arenas.