I am guessing you didn't do enough full rests at the beginning of the game. My character slept with him before even getting to the first little groups of goblins, way before the party. If it's too late for you I don't know. Do your best to raise approval. If you don't like making some of the choices he likes in conversation, pick them anyway then do whatever you want afterwards. Doing a good playthrough with him is fun.