Any speed value on a weapon is subtracted from your recuperation time, so higher is better (though you can not get the total below zero). At least with swords, the weapon expertise is going to reduce recuperation time to zero anyway, so it is a little helpful in the beginning of the game, but not a significant attribute overall.
I went primarily with damage for swords and then looked at bonuses; stat or hit point bonuses were best, IMO, followed by damage spells, but speed was never a consideration. Even the speed difference switching from a one to two handed sword wasn't a huge effect, though I eventually switched back to a one handed sword (not quite as much damage, but a little faster and more bonuses from equipping a shield).