You still have racial traits, and all the social baggage that comes with being a member of your race in whatever setting you're playing in. That's adversity you don't need mechanical bonuses to represent. Then there's the social stigma of being say, a super smart orc from a culture where that sort of intelligence isn't valued. Or an elf who's viewed as bizarre and unsavory because of the focus she puts into honing herself for strength rather than the nimbleness that's seen as desireable. From my perspective, the adversity that comes with not getting a favorable ability score is just the least interesting kind of adversity, because it simply means that you're mechanically worse off than another option. It also denies you the ability to be BAD at something your race is seen as good at and roleplaying the results. Floating ability scores mean you can be a weak orc who had to make up for it by being clever or cunning, a tongue-tied half-elf who can't quite say the right thing when he needs to, etc.
