You are correct; bonuses from melee / ranged weapons only apply when you are using the respective weapon. However, you can equip a shield when using unarmed combat, and the bonuses will be applied with the shield remaining on your character's back.