I had the same problem as you. In the end Tarquin was grateful to me for being a great friend(I wasn't, nor was he a 'friend). So you'd probably miss out on a quest but don't think it changes things drasticly.