The issue would of course become 'Whoops I just destroyed the only way out of the cave, and all of my characters who could otherwise leave are dead'
Well that's an extreme. Having destructive structure doesn't mean everything can be destroyed.
It could be fun to ha encounters with destructible parts.
Archer on a ledge? destroy the pillar supporting it.
Why not but it would still be very restricted.
Might still be fun though.