If you have issues with Withers not in the game then play easier difficulty. Same with nerfing special arrows, potions, camp supplies etc.
Give the tourists a tourist mode, which they already have “explorer “ and give the actual players who likes the satisfaction of beating a difficult encounter their own mode.
I am currently in Act 3 on my third playthrough, my second on tactician mode (first playthrough was balanced). This playthrough is the first time I ever used Withers and respec, at level 2 I respeced Shadowheart to War domain. Not to make it easier, but just to experience another subclass as I am never going to play a Cleric as my main. Just like I have done at level 3 with my other party NPCs, that I am unlikely to ever play as a main. This play through Astarion is an Assassin previously he has been Arcane Trickster and Thief and I have played at least 2 of the possible subclasses for the other NPCs, Paladin excluded.
Respec is a quality of life feature and that is exactly how I have used it respecing Shadowheart to War domain. Because Clerics and Paladins choose their subclass at creation level 1 unlike the rest of the NPC classes you can party with who choose their subclass at level 3. Sure respec in tactician mode can be used as you have done in your impressive solo videos, but why would I with a party of four when I find tactician mode beatable without any form of min/maxing.
I never multi classed or tadpoled, I not even optimised my main's specs and I have completed tactician without resorting to respecing*. So removing Withers would not increase my satisfaction of beating a difficult encounter.
I actually have no problem with not being able to respec it should be an additional selectable choice to the difficulty modes and available at release. RPGs of old and some even today have a lot more options than just 3 difficulties the difficulties could be fine tuned such as choose a difficulty and add some options from the higher difficulty. Along with additional options such as Ironman, perma death. I have a feeling that this new difficulty mode will be something like Ironman such as single save perma death for party wipe and no respecs.
But if that's the case it should have been there at release and it would not make encounters more challenging which is what is needed.
*To be honest not even optimising the party unless their presence was required to complete a quest or I was forced to use someone else (Orin kidnaping Lae'zel) my party was always Ranger, Lae'zel, Karlach and Astarion.