I don't really like the idea of weapons dealing "magic damage" and vice versa.
This is where it gets kind of weird having weapons that deal elemental damage. If I cast a magic fireball, I assume the damage is magical. If my sword deals fire damage, is that magic damage? What if I throw a napalm grenade? What if I walk into a fire?
How is physical vs magical defined?