Generally speaking, each character should specialize in either Physical or Magical damage, because as you realized, it is usually a waste of time trying to get down two armors at once with the same character.
Hybrids in this game are generally a physical damage dealer picking buff and utility skills which don't require points into INT and using them to complement and aid them.
