I've also recently picked up the Enemy of Justice tag in the goblin camp.
The game seems to treat some of the goblins as temporarily hostile guards (perhaps ones summoned by drums?) and it may have been killing one of these that caused the problem.
All I've done since is go back to Waukeen's Rest and then back to the Druid Grove, and noone there has been hostile so I'm just going to continue to see what happens. But that does suggest that your problem may not entirely be due to the tag as I've not got the same effects.
It may be worth submitting your save file with a bug report to Larian if you haven't already. There's a "Report a Bug" link up along the top of the game launcher that'll take you to the right place on their support site.