I think restricting memory is not the correct way to increase difficulty. It is self regulating. The more times I go through the game, the more I tend to reduce the amount of spells I use - HOWEVER it is a very fun aspect to have more versatility vs a slightly accelerated combat stat. I think this falls under the broad category of don't fix what isn't broken.
Basically if someone wants to have 30 skills and be all over the place - why not let them? Is it the best way to build (probably not) but I don't see the benefit of artificially restricting someone from a bad build. Furthermore, restricting down to very low memory slots would benefit some builds over others and mess with balance which really isn't ideal either.
