Just completed this quest - yes knocking down the wall from the south side jumping in and then grabbing him is the trick. He fell down again and didn’t follow me but I jumped back in and second time he was fine and followed me out. I did notice the fire seemed to barely come near us which was a first in many many tries ….
