I think it's caused by NPC's getting stuck with their pathfinding. This usually happens after they run away from combat.
The elf by the tidepool with the geo spells for instance can get stuck on the rocks, and for me, one of the quest givers ran off and got stuck behind a wall, the issue resolved itself after I teleported or killed the NPCs, so I'm pretty sure their broken pathfinding was causing the lag.
