I love the game, but I am surprised that systems as big as armor were not balanced better.
Yeah thinking about armor got I this idea:
Items still give magic or physical armor but you only have a single armor pool(the sum of both) BUT the vulnerability of that armor pool depends on the the phys/magic armor ration. If 200 of your armor comes from magical armor and only 100 comes from physical armor then would physical attacks deal double damage.
You can still specialize in one type of armor but mixed parties are competitive again.