No, no - I didn't break through the wall. I used the lever in the Warden's office before giving them their hammer. If you do that, they jsut run out towards the docks. Once outside, you have to protect them, which is very easy.
After that, they just stand in one spot, not interactable and if you leave the area, they die.
That was the solution. Failed quest
