A staff does the damage listed (of that elemental type) in melee, or 80% of said damage at range using the Staff of Magus skill. A staff can also hold magical enchantments just like any other item. That is all a staff does. If you want to use another weapon you can do so with no penalties to spellcasting.
Robes are things you wear. They can hold magical enchantments just like any other item. That is all a robe does. If you want to wear plate armour, you can do so with no penalties to spellcasting. The only reason why you might want to choose otherwise is because heavier armour is heavier.
It's just that simple.
