yeah, that, the complexity like you said and for balance reasons. if they didn't remove the skill / upgrade points when unlearning the summoning doll skills (if that were possible) you'd be able to forever increase them. sort of spoiler like. hence.