I got in the cave sparring all huts and towers (saving them when going to the hall of echoes for max. XP, since Dragon Targets never downscale XP) and he still mentioned that to me.