So what I think memory should do is focusing a character's range of possiblities to the abilities the character's good in. It shouldn't serve as a rigid limit to only have a very small amount of spells at hand, especially in late game. I think that Larian should give the player more freedom not less. Memory is good to balance the game, but again it works together with AP costs and cooldown costs for skills. Less memory requirements don't give you a free pass on spamming high class spells. It gives you the chance to make full use of the possibilites a high score in certain skill class or magic schools should give you and it makes the game more fun by giviny you a greater range of spells you can actually use in combat encounters.
I will conceed that what you write makes some sense.
However, in it's raw form, your idea provides multiplicative benefits, which would quickly make all skills only ever cost 1 memory for characters who choose the corresponding skilltrees, and put a severe toll on hybridization.
Operating on the player's capacity is much safer in that regard, because your primary means of expanding the number of skills is already to expand the number of slots.
Adding to that is precisely that, 'adding to' not multiplying.
2 points lost = 1 memory point for below 10.
Done... It is never a factor except in that RARE situation... (In fact... How do you lose memory?)
That's an inconsistency of rules, inconsistency is bad design.
IIRC you lose memory from the Ring of Braccus Rex(-1), and there are hopefully going to be other items with similar trade-offs.
