Well per 5e rules the Pact of the Blade boon only applies to one weapon, therefore you should only be allowed to use CHA on which ever weapon you have designated as your pact weapon. It's generally not an overall affect on your character, but on the weapon itself.

Also Larian obviously could have changed this completely, but that's how it works in 5e.