You don't have to avoid using it, just go out of range when you break the concentration on it, and after a minute, you can again normally approach the NPC and they won't be aggressive.