Just to mention, the line in the game isn't "I met a man who wasn't there."
The line is, "I met a man who wasn't a man."
Here is the quote from the mason's log book:
- Sick of standing idle while Justiciars gain power in our humble town.  What will become of us if we allow it?  I met a man who wasn't a man.  Touched by a devil.  Or maybe worse.  But he offered me something I couldn't refuse - help.
This is, I believe, a reference to Raphael.  The stonemason made a pact with Raphael for help.  That's why Ketheric was defeated a hundred years ago.  That's why Grymforge became what it is.  Yurgir and his army of merrigons invaded.  Yurgir had a quest to get rid of every last justiciar.  Of course, Raphael also made a deal with the last justiciar, splitting him into numerous rats so that Yurgir would be forever stuck trying to finish his task.