You may have used the Friends spell a bit earlier. After the spell wears off, the subject knows they were coerced with magic, thus the comment about upsetting the guards.