Yeah, I'd scrap all weapons passives and just toss it into the Skill Sets.
Man at Arms: Increases 1H/2H dmg by 5% per point
Expert Marksman: Increases Bow & Crossbow Dmg by 5% per point
Scoundrel: Increases daggers damage by 5% per point

Mages should get bonus damage to specific magic elements.
Geomancy: +5% earth dmg etc.