I was in the same boat the OP until I read about not mixing Magic and Physical damage. The game went from frustrating where I was cheesing every fight to kinda easy. I was also surprised at how nasty some of the posts here are so I thought I'd share too.
I do miss having a balanced party for loot drops, play style and feel of a traditional RPG group. I have magic users with high finesse and strength so I can use gear drops, and running around with 4 wand / shield spell casters is plain goofy.
I love the game, but I am surprised that systems as big as armor were not balanced better.
I think you can still have both, each build can dabble in areas so where they are stronger in one form of armor, they can at least revert to their secondary choice of damage and be reasonable. There are many enemies where they are strong/weak in each armor. Being able to attack either way also makes the game easier. Yeah what you don't want is a character that cannot due physical or magical damage, but why would one build so narrow?
