Tbh after I respecced Astarion at the beginning because I AM THE ROGUE not him, to be a ranger, I forgot the function existed, I also forgot Withers existed and never talked to him for the rest of the game because he didn't have anything interesting to say.
I can't really see the reason to nerf an optional feature in a single player game. If this game was in anyway competitive I would say go for it, or if they had a deathless run, maybe put that in. But as it stands theres no reason you can't just forget Withers exists.