Gandalf is a staff user not a wand user? That's why he normally either the staff or the sword, but not both at the same time?

Going for magic and physical armor at the same timme will also never be really effective.