In my opinion, it would be a pretty good compromise to increase the x2 modifiers.
It would be an extremely simple solution that would make a character who specializes in something less often fail.
There would still be a possibility of failure but I think the game would be generally more fun for more players.
Thanks to this, the player would feel that character building really matters, not just rng.