You definitely don't need to visit ALL the Source Masters, I never talked to Ryker or whoever was in the Blackpits (never went).