Also don't talk to certain npcs about your romance with him. Your character will apparently even agree when they call him "boring".