You need to meet him again after getting the fortune (separate from the shrine encounters).