Much less of a pain if you use a crushing weapon or Eldritch Blast. Lae'zel with a 2H hammer can smash the (ground-level) crystals in one hit.

The really annoying ones are the crystals on the stalactites, which if you don't have a Warlock you have to hit with a bow.