Chromosomal determination of sex is absolute and there is no ambiguity there. Males have the XY pair of genes, and females have XX. Some males may have XYY, rarely, and there are even more rare cases of true hermaphrodites which are the third classification. Natural physical characteristics are derived from the individual's set of chromosomes
Just as absolute as it is ultimately irrelevant when we're talking about entire multicellular organisms. There are XY individuals who are phenotypically female and capable of giving birth, for example (Swyer syndrome and a few others), just as there are XX individuals who are phenotypically male for a variety of reasons, the most obvious being SRY translocation. These individuals would not be called "true hermaphrodites," since they don't have both ovarian and testicular tissue. The actual "true hermaphrodites"* are not their own classification, either; despite having both XX and XY germ cell populations and corresponding oocyte/sperm progenitors, they are not scientifically or medically categorized as a third sex, nor are they capable of reproducing as both males and females. They are, however, "intersex," and this term covers many different disorders, many if not most of which have nothing to do with chromosomal sex.
"Natural" (is this an appeal to nature or just an unfortunate choice of words?) physical characteristics are only derived from a "set of chromosomes" if you're going for a massive and, frankly, incorrect oversimplification. What they are actually derived from is protein activity, which is determined by many different factors (maternal transcript and hormones, exogenous endocrine signaling, and, yes, gene expression of the body itself
subject to epigenetic modification, mutation, and epistasis). Even at its most basic level, embryonic masculinzation (i.e. Mullerian structure regression and testicular development) is not guided by the entire Y chromosome but solely by its SRY gene, which is a transcription factor upregulating
SOX9;
SOX9 is found on the autosomal (non-sex) chromosome 17, and an activating mutation will cause its expression and subsequent masculinization of a fetus without any involvement from SRY. Vice versa for
NR0B1, whose overexpression will negate SRY presence in an XY fetus. And this is, of course, just the very tip of the pathway.
You can, of course, say that an XY lactating mother is chromosomally male, but outside of a few specific contexts it is a vastly pointless observation. For nearly all intents and purposes she is female, both socially and biologically and with corresponding medical risks.
__________________
* - the term "hermaphrodite" is typically used only for species where it is a natural reproductive state, and using it to refer to people is only slightly less outdated than calling someone with an intellectual disability "retarded."