Honestly... Just keep the Memory stat at 10... Give the player only 5 memory slots to start.

And every 1 memory point from then on = 1 memory slot

It might be ugly but I honestly don't think people will mind.