I'm currently playing through with an unarmed character and it is indeed viable. One of the reasons for this is that your base damage automatically increases each time you level up - even without putting points into Strength.
Single-handed theoretically attacks faster than Sword and Shield, but the speed of attacks isn't really a huge deal in this game.
Your Bow uses Dexterity as the primary stat for determining damage, melee attacks use Strength, which accounts for the difference when switching weapons.
