It sort of was built by one family. Underdark societies tend to be close knit and survivalist. Svirfneblin live in intense, crowded conditions. A mining expedition would probably live, work, sleep and eat right next to each other. They live cheek by jowl and are obsessive about their work. Gems! Svirfs love them like a Dragon loves it's hoard. Physical or emotional discomfort is nothing compared to finding and working gems.
Calling it a village is misleading, it's a mining camp. They support themselves with fishing and foraging (not growing or raising), and construct with what comes to hand. There are no trees - that's an important part of world-building in the Underdark - so other resources come to the fore. It's a camp - essentially temporary in nature.
It's is an old camp too. I'm sure I remember one of the journals mentioning that this was an older excavation, and they've found that it wasn't mined out after all.
