This issue has been touched on before a couple of times. Memory stat is there as a form of balancing. Without it players could literally learn and field every skill and spell in game, you'll have every counter on hand and your character is basically a 1 man army making having party members pointless. Even D:OS/EE system is pretty much broken as ultimately, you can field all or majority of the skills & spells by mid level.