+1 i don't go past 20 memory because my hotbar anyway can't hold more than a certain amount of skills.
Alternatively, in the first Divinity: OS i remember using the skills directly from the skillbook but that doesn't seem to be possible (skill needs to be placed in the hotbar) in this game.