It's a puzzle - so you can assume that all relevant information is in the riddle:
Well, it is not that kind of puzzle. Asking for more information is part of the game. I'll take your statements as questions, okay?
Q: A man - so it might make a difference, that it's not a woman. -
A: No, it might as well have been a woman.
Q: comes home - so it's relevant that it is in his home, not his working place or elsewhere
A: Yes!
Q: does NOT notice the saw dust - so it would have made a difference, if he had noticed it?
A: Yes again!
Uebereil's questions:
Q: Did he work with wood?
A: No
Q: Is it important?
A: Yes
Q: Is his work all that's important?
A: No
