The problem is that you are mixing things up. Strength is not only how strong is your character, but also how often he can hit, how much damage he inflicts, but also how much he can carry and how it can interact with other creatures.
An halfling is already "weaker" than an orc, because of its size that gives him disadvantage interacting with creatures bigger than him and making him carry and lift less (IIRC). Why do you want to make it also a worse swordman?
