Well it's officially a huge PITA.
I started back about 10 hours and fought my way back up using a slightly different build. Made it through the source temple, got restocked and decided to finish up some quests before the ending battle.
Soon as I 'warped' back to the phantom forest? Problem came back full force. I can go out for a beer and pizza and the guys will still be s l o w l y....... doing their turned based attacks against me.
As for area spells? So far totally dead and useless. Just did the meteor spell to kick Imal down a few pegs. Took the spell and the points to do it.....No effect at all. Not even a sound.