For me, I got to the Graveyard and he wasn't there, then warped back to the Lady Vengeance and he suddenly appeared next to me and said hello. I haven't been given this quest, and only found out about it after finding some sword pieces and looking online.
Not sure what triggered his return to the boat. When I try to talk to him whilst holding the sword pieces, he doesn't bring them up in dialog.