its the memory-eaters fighters that are the ones killing you. if you notice your health goes down by about half when one of their orbs hits. the easy way: avoid the orbs and summon the wyvern to distract them. then when the coast is clear go in for the fire breath on the towers
