I distinctly remember when I first got the game after launch, my first character, who was a Bard, spawned with a Rapier. I recently started toying with the game a little again for the first time in over a year and discovered this bug. Looking to see if it was happening to others, I found this post but no answers, so I did some light modding to try and figure out what's going on. Removing the Hand Crossbow from the Bard's starting equipment list will make them spawn with the Rapier (but, obviously, not the hand crossbow). Furthermore, adding a second Rapier to the starting equipment list will make them spawn with one in their inventory, but the one that's supposed to spawn equipped will, once again, be gone. Changing the order of the starting list, or trying to remove items other than the Hand Crossbow do NOT fix it, nor does changing things like making the Rapier sheath on the hip instead of the back.
What I THINK is happening is that since the Bard gets to pick a piece of gear during character creation (their instrument), something about the way the game loads new characters is causing that instrument to replace the Rapier when the character is actually built.