You are not as limited in points as you think. Once you unlock the Hall of Secrets in your Homestead, you can buy two books that give +1 Ability Score and +3 Skill Points, respectively. Every time you gain a level, these books will restock.