hm either it's down to the type of damage ( fire / earth / air / water would then be magical while Slashing / Piercing / Crushing would be physical ), or the type of attack or even the school of the skill.
I can see where you're coming from though. Wands fall into a weird category. My guts tell me it shouldn't work together, Bully shouldn't affect the Wands damage, but it may require a few tests indeed.