In my experience Colossus Slayer works pretty consistently as long as you don't use Hunter's Mark. Still wish I'd gone Horde Breaker, but finding this sort of thing is what early access is about. I hope they fix it before too too long, but I imagine game breaking bugs probably take priority.