I actually had the gems on me when I talked to him the first time but he didn't notice then so I figured I had to go get the gems someplace so I went to that cave with the Patriarch and got the gems, came back, and he still didn't notice. Of course! He's blind! Well, that's the best answer I can think of anyway.
