Within reason, you should unlearn all skills whenever you want to make at least moderate changes. You don't want to try making frequent changes, but as long as you have a good idea of how you want to adjust your character build, and are either pretty sure you will not need to make additional changes or you don't have the spare skill points to easily put off reskilling, go ahead.

The cost to unlearn skills doubles each time you do so, but as gold is relatively easy to come by you don't have to worry too much about that the first couple times. I reskilled right after getting the battle tower, and then again before heading to the hall of echoes, with enough gold left over I could have done so a third time (or forth, with a bit of malachite ore / diamond farming).