I have both had and not had this problem. I'm not sure what determines whether the broken drum can be used; I know that if I break the drum with an arrow, Goblins will still run to and use it as if it were just fine. If I break it with something like Alchemist's fire, goblins won't return to and use it even after the hazardous terrain recedes.