Yes, it would be a shame but I feel that's the way it is. In my current run I pretty much forgot about him until his quest became relevant, took him along and sure enough, his affections stirred. But, on a positive note, his dialogue now registers if you didn't ask about his love life and you can politely reject him without any awkwardness. Seems banishing shadows just is his love language. (If any dev reads this: thank you for the fix!)
