Only way I found to stop them is to body block the AI. If you set up 3 of your party to surround the drum the AI goblin that would go beat it up there will get body blocked by your party and then just start attacking.