There are a couple small places in the game where you can not use magic, though in the first you don't really need to get involved in the combat, and in the second you can avoid it (there are shadow potions nearby, along with some other potions to loot, you can use to bypass a group of hostile creatures).
At least with swords the expertise skill eventually drops your recuperation time down to zero, anyway, so the minor differences between swords become irrelevant. I don't know if that is the case with bows; level 5 Bow Expertise drops recuperation by 15, while Crossbow Expertise drops it by 5. You can also get a Speed bonus on weapons, which also reduces the recuperation time (though the net value can not get less than zero).
