It doesn't make physical sense, but from a gameplay perspective it would be brutal to have to match gender and race for armor.
I don't know. It doesn't seem like it'd be so hard to have small armors and regular sized armors. Little things like that would go a long way, imo.