That's right: the difficulty of a standard Drow patrol/raiding party ought to begin at CR 3.