Say, the father who is building a house for his family might be building it from sacred wood, or maybe on some elven burial site.
You're changing the scenario. The point is that these situations are unambiguous. It doesn't matter if you're an elf, dwarf, or whatever. A father trying to build a house for his family is unambiguously a good act.
Oh, but what if he's a vampire! And his family is brain-eating zombies! Well, yeah, with that knowledge you might be justified when you decide to paint their house red.
But that doesn't mean you're justified killing everyone on sight, little old ladies included, simply based on the logic that they must have done
something to deserve it.