It may take a little while, but it is not impossible to completely clear the starting area in the hall of echoes (at least not if you are about level 34 or more). Once you take out the nests it is just a matter of flying around using Fire Sphere to defeat the memory eaters.
I didn't bother with the shield spell, either.
