Yes, the stat system as it currently stands is.........quite horrible to be honest -_- On the other hand, I'm surprised that I find myself liking Memory in order to make one decide in how specialized they want to be. Oh, and to be quite honest, I'd love it if they replaced crit chance on Wit with something else; there's so many ways to ensure guaranteed crit or build a high crit chance that I don't see a point...thus making the stat seem sorta wasteful if all your getting from it is initiative and chance to find secrets easier.
You like memory FOR NOW... since all the skills (for the most part) cost 1 memory.
Wait until the 3-10 cost skills.