Honestly, just make pact of the blade have charisma to attack and damage built in. You've fixed blade and removed the need for hexblade.

Armor wise, you could still require investment into feats, or you could make medium armor baseline-there are already several companions whom use medium armor though and there isn't any good heavy armor in the current build, so IDK.