You need a human fisher's village to have Tieflings, Mind Flayers and Elves be something special. If you continuously throw powerful races at us, they become boring and dull.
I get your point and I agree that fisher's village would make illithids look more special but on the other hand I started in far too many "fisher's villages" to enjoy them even a little. Even tiefling refugees seemed too generic for me.
