Choosing, within the limits of the plausible and with secondary priorities in mind should they exist, ability scores that fit a character's primary competence, is just playing the game with a measure of competence. Not min-maxing.
The racial ASIs are a different matter though. These games work from the principle that being an outlier is more expensive than being average. And the racial ASIs tell us something about the world, namely that the baseline average of a race is different, so a high INT half-orc is more of an outlier than a high-INT human and thus more expensive. The problematic part, from my point of view, is that there seems to be - I don't know this part of the rules, actually - a limit to the level above the average you can buy, regardless of how many points you spend. So that it is actually impossible to make a high-INT half-orc rather than expensive. That I consider undesirable, but "more expensive" is OK. It just reflects that you're more of an outlier. So I'd just remove that attribute limit for everyone. If you want to set everything to 8 and buy INT 18, fine.
Beyond that, unless you want to overhaul the system and remove any extra cost for being an outlier for *everyone*, you can't keep a race's average different from any other's without it also being more expensive to be a non-typically competent member of that race. So what do you want? Do you want to compromise the integrity of your races in terms of worldbuilding in order to make it easier to be a non-typically competent member of a race, or do you want to keep your races intact in terms of worldbuilding but also keep it more expensive to be a good half-orc wizard than a good human wizard. You can't have both. I will almost always prioritize worldbuilding. Unless you really want all races to have the same average ability scores - and why the heck would you ever want that? - you shouldn't remove racial ASIs in character creation.
I think the thing that gets in my craw mechanically is that as you point out, a non-standard wizard will always be behind from the start of a campaign to pretty much the end, which few campaigns reach anyway. They'll always be behind any party members that do play into a more typical race-class combo. But another thing, I also brought up the example of a dwarven wizard, and I did that for a specific reason - why is that concept such an outlier? Or a dwarven bard? Or an orcish bard? Why is a tiefling fighter an outlier at all? Or a tiefling wizard? Or a tiefling barbarian for that matter? Tieflings can be born into any culture or family, why wouldn't you get a barbarian tiefling that's really good at Barbarian stuff? A halfling cleric? Why can't halflings be just as devout and mystically powerful as any other race? What about a gnomish rogue? What makes them an outlier? When you actually stop and break it down, the 'outlier' combinations become more prevalent than the 'typical' ones, or very near to it.
The "outlier" characters as you call them can be just as powerful and effective than the more common race and class combinations. Do they reach their potential as soon? No, but I don't think they should. They are outliers after all. Yes, you will have fewer feats (if your group plays with them), but that is the price you have to pay.
You don't need to have an 18 in your primary stat at level 4 to be able to play, it's simply not necessary.
