You have to talk to him with one of your two main characters. If you're using Madora as an active character when you speak with him, that may be why he doesn't want to talk.