I've gone throught all the talk options, including the threatening one a few times, and then left to the South or to the West.
After a while, I go back in, but the guard is still there!