Chaucer wrote something about this in The Miller's Tale:

"Abak he stirte, and thoughte it was amys,
For wel he wiste a womman hath no berd.
He felte a thyng al rough and long yherd,
And seyde, "Fy! allas! what have I do?"

Females can certainly have facial hair, though many would prefer not, but I have yet to see any display the "mutton chops" of a proper gentleman.