Because casters, by and large, are going to be very vulnerable at lower levels due to their typically low HP and armor values.
In D&D particularly, you don't want to be the target of a volley of arrows before you're aware some elves were tracking you, etc etc. People just don't prioritize killing civilians unless it's a clear VIP that affects the attackers, since the civilian is no threat.
I suppose it's possible they might if they're simply wildly and randomly choosing a target, but that's hardly the norm to encounter that kind of hatred in the game.
I guess it's prudent to have the surprise factor as an additional weapon as a caster who typically needs spacing.
In D:OS I don't particularly think casters are that weak, even at close range, which I think is a bit of a problem, but I have yet to truly figure out what good balance would be for the game as I have been more worried about other aspects.
