I think that if they made dexterity reduce the amount of action points it takes to switch weapons, it would be perfect. Dexterity characters(bow/dagger) are essentially the only ones I can see needing to switch in combat. The other solid option is making daggers a "quick" switch. You can take a dagger out for one or maybe no AP, but taking your primary weapon out would still be the normal AP cost.
