Sounds like you hit a bug. Fire Sphere should pretty much be a one hit kill for Memory Eaters, which makes it relatively easy to fly around and clear them out faster than the nests can produce more.
Did you try returning to the battle tower, and then back to the Hall of Echoes? A couple people ran into particular opponents that were invulnerable (ie the goblin shaman near the start of the fjords by the Grand Knight teleporter shrine), but leaving the area and returning, or doing something else on the same map and coming back would usually fix the problem.
Try emailing support@larian.com, with a description of the problem, and maybe a link to this topic.
