Just a couple of off-the-top-of-my-head, probably bad ideas:

Maybe instead of armour getting removed if your stats become too low to use it, instead its protection is reduced to 0 for the duration of the lowered stats?

If your stats become too low to equip a weapon, perhaps instead it increases the AP costs of all normal and weapon-using attacks by 2 AP?