Tip: when playing as unarmed fighter, equip a shield. You do not get penalties. The shield will hang on your back, but you will gain from its stats (armor, enchantments, charms...).