Memory is really only needed on Mages
Seriously depends on how far you are doing the split.
If are just a necromancy + Whatever mix... then yeah.
Outside the source spells, the amount of useful in-combat spells are quite limited if you don't grab some int for more damage. I can see going Pyro for Haste and Aero for Teleport on physical characters but that's pretty much all you need because at the moment, you need a double mage team to be able to break magic armor fast enough to be able to inflict magical CC.
And most source spells on physical character aren't even that great either. I thought Split arrow massive amount of hits was going to be great from the description. Boy, was I disappointed.
You can get 4 Mems from Belt, Necklace and 2 rings and that's enough to last until the end of Act 1 until more Level 3 skills like Chain lightning are released but by then, our current save will no longer be compatiable.
