I think one way they could mitigate this a bit, is to say people like this were intentionally gathered for some reason or another, that the fact they are all a bunch of weirdos is not a coincidence.
Or maybe mindflayers are just considerate and only kidnap those that no one will miss?
