Yes when he first ask you do the quest you can decline it.
I met back up with him to help him fight Goblins now he dissapered so I can't see If I could give him the hearts.
The Keys I have are related to Quests but I dont know what quests thats why Larian should put some more details.
It kinda bugs me for some reason.
I upgraded my inv size so I have more room