I think that would only be a concern with Devotion paladins, and even then it depends on your interpretation of the idea of Honor. One could argue that if you're honest about your intentions, treat your casaual liasons well and don't just treat them as knotches in your belt, or objects there to provide you pleasure and satisfaction, then no, you're not breaking your vow.
