Every point into Memory is one fewer point into more damage. BUT sometimes the right utility skill can save a fight for you that you would otherwise lose. So It's a gut decision, really.
Take a good look at your complete list of skills. Are there skills you have memorized which you don't use? Unmemorize those. Are there skills you don't have memorized you think "I really need this"? Memorize that. Make a round of the skill vendors and see if they have any skills you think you could use?
After looking at all your possible skills you still think "I definitely need more skills", then add a couple points. I have a (probably bad) habit of not allocating 2 of my attribute points just in case I decide I need more Memory slots right now.
