In fact Cross Classing is the way to go now. Even if you don't invest in other attributes... A Fossil strike, fortitude, fireball, magic armor... will still be nice.
I am always curious why people cross-class only mages with other mages. Are there only mages in the game? Is it not possible to cross-class Berserk vs Duelist ?
It is because the stat mentality is hard to break, in fact if my friend didn't invest in magic anyway I wouldn't have noticed.
Sure he did 25% less damage than me, but that was still a good hit when we needed it along with some much needed CC (Necromancy is severely under-estimated... Chains and Decay are deadly).
While Mage with Mage gets the full stat benefit.
---
Now why the physical skills aren't combined more often is a bit more simple... They are MOSTLY incompatible without the proper weapon.
Most Marksmen skills don't work (Only two, the teleport and First Aid)... Scoundrel relies on knives for two or more of them.
While Warrior doesn't NEED a compatible weapon, a lot of the skills are outright damage dealers. Though his berserk skill is incompatible with backstab (the main source of Scoundrel damage).
Marksmen has the most compatibility issues pretty much period... at least currently.
While Warrior/Warfare has the least but is a very up close and personal skillset.