You don't get them back, but it is worth it. Its only 3 points, and latter in the game when your skills are over 100, one point in each skill no longer seems that important.