I decided to ignore him at the camp when he has an exclamation mark above his head. I just wonder how long he can go without consuming an artefact?
You can talk to him and just refuse to give an item, he will complain but nothing happens. For now at least.