I didn't really find it annoying, because I built my characters carefully using the guidance of a FAQ on skills, but the lack of a stat respec definitely makes it harder to change the class of your character.
But starting a new game offers some new advantages, you can try out completely different solutions to some quests.
