The armor system took some getting used to for me as well, but now, after having played a few hundred hours, I grew quite fond of it. To me the distiction between physical and magic armor adds more depth to the combat system, just as the terrain interaction capabilities, the teleport skills, the elemental interactions, etc. It plays its part in combat not becoming mindless bashing, which is certainly the case in many other RPGs. Well, at least not in the first half of the game, anyway.

So on the contrary, I'd say the armor system is a nice unique idea, and it makes you think during combat. Which to me is outright positive.