Ah, it's not something I had seen before; certainly, I would expect NPC "owned" crystals to be flagged as such. There are so many inconsistent behaviours in EA that I do begin to wonder whether Larian will actually be able to address them all before the full game launch, or whether we will have to rely on the usual "unnoficial fix mod" that so many games seem to need.
in this patch, unlike the other crystals, the two crystals near the entrance are definitely outlined red when targeted. it is also possible that destroying something in a settlement is seen as a bad thing but I don't recall being told off when I destroyed my tower of crates in the middle of the druid grove so it is unlikely.
as for the unofficial patch thing, larian isn't bethesda, the only way for them to leave ugly bugs in the game is to go bankrupt trying to fix the others bugs...