If going full magical, I recommend having at least one character adding enough STR to use physical armour and one with enough FIN for rogue armour, that way you're not wasting good drops, especially ones that add to magical skills!
That is a good idea, because it only takes 4 points (for a total of 14) to wear the best armor of another type. In fact if you don't, by the end-game, you'll have a notable weakness to physical attacks. It's splitting points evenly between two primary attributes which is problematic.