Do most people build their party around doing either entirely armor damage/entirely magic damage vs. having a mix of damage types?
I don't think so. Those who play this game exactly one time most likely don't go for the "either or" 100% one damage type. Those who play through the game multiple times probably do that a couple times, I'd guess.
Having only one magical damage dealer in the party is not so bad. That's actually what I'm doing on my current playthrough. I must say I'm actually having an easier time than on my first playthrough in which I had a 2/2 comp, with more optimized builds. You just need to make sure the magical damage dealer does A LOT of damage. Right now I'm in the last portion of the game; only my main does magical damage, and she mains Geo and Pyro.
That being said, I have a tank summoner who can have the summon deal elemental damage in case I need it.