Would there be any merit in taking a level in Draconic Sorcerer (White Dragon Bloodline) with a Swords Bard instead? Would give access to the Shield Spell and Armour of Agathys early on, which would both seem to be good things for a Swords Bard to have, as well as a selection of melee and ranged damage spells and cantrips.
Also if taken at level 1 would give you the bonus to Con saves (would need to play as a Drow to get Rapier, Short Sword and Hand Crossbow proficiency in this case, though that was going to be my racial choice anyway), which would help a lot with concentration checks, though obviously would delay your Bard progression from the off (and I don't want to mess around with respect, if I can avoid it).
Any trying to decide between just playing a single-class Swords Bard or Draconic Sorc 1/Bard 11 or a (Dex-based) Paladin 2/Bard 10 - the latter can certainly do more melee damage, but I'm not so keen on the idea of playing a Paladin (for RP reasons and concerns about maintaining an Oath), nor on using up a load of my spell slots on Smites instead of casting spells. Have even considered Ranger 2 / Swords Bard 10 (not necessarily in that order).
Actually that very thing was suggested to me on Reddit, but for this guy I want Pact of Blade... as much for RP as anything else.
Based on that Reddit advice I plan to make an 11 Bard, 1 Sorc, starting with Sorc/White Dragon, for the Armor of Agathys, and a more damaging cantrip. He'll be College of Lore though.