The theory I've heard is that the game is checking for adjacent-allies within 5-ft of the center of enemy models. Since medium enemies have a 2.5-ft radius, this actually means your ally needs to be within 2.5 ft of the enemy. And it's impossible to be counted as adjacent (for the purposes of sneak attack) to a large or bigger enemy since their own character models take up space equal to or greater than this 5-ft "adjacent-ally" radius.
This would explain my polar-bear-druid not being able to set up sneak attacks with Astarion. Maybe it also checks the center of the player-controlled character model?