You should best click a good distance above the "door" (so that he/it is not highlighted), as if you wanted to an area far away.