I think it could be fun, depending on the loot.
Damaging potions, scrolls, or other fragile things, that sounds logical.
Damaging a warhammer with a fireball feels a bit weird.