There is no explicit warning about not being able to return from the Hall of Echoes, but the dragon Patriarch did hint things didn't go well for Maxos, the only other person able to enter alive.
At level 36 (1265 hit points, with the +40% HP and +40% offense armour pieces equipped) I was pretty much invulnerable to the level 30 memory eaters. At level 35 you should be fine as well, not getting one shot killed; presumably whatever messed up in that save effected your resistances, or something.
The first time I went through that area I think I did take some damage attacking the nests, but later reloading a save to check something, I ended up siting next to a commander in the middle of a group of memory eaters getting regular attacks and the occasional Fire Sphere setting me on fire, but never took any damage.
