I suggest Paladin 2 - Bard 10 if you really want to play a swords bard that can crush things. You end up with the same amount of skills, just fewer choices for the fist two. However, you gain a lot for picking Paladin 2 first including the early shield and medium armor, which gets you to 19 AC in the first hour and a half of the game.
Getting the smites and dueling at level 2 will pay off dividends if you rest frequently as each hit counts for almost three hits. By the time you would get multi-attack by staring Bard, you will have two 3rd level spell slots uses of smite, which is going to put out way more damage when you hit than a straight Swords Bard could do.
If you are only worried about damage output, I wouldn't go back to Paladin ever.
My one caveat for this build is that I would suggest you respec for one single reason at level 5. I'd change your fighting style for Paladin from Dueling to Defense or Protection and pick Dueling up from the Swords Bard choice. Nothing really changes in you character that way as you have all the same things you did up to that point, but you dont get stuck with Two-Weapon Fighting. Up to you of course, but to my mind that is just fixing a limitation rather than breaking immersion.
