I've yet to see Colossus Slayer work, and I regret taking that instead of Horde Breaker or even going Beast Master for a bear or wolf.
It only works for me when I DON'T use Hunter's Mark. And even then only sometimes (I know the target should have less than max HP, so it's not that).