You're basically talking about some sort of morale system. If they fail the morale break they'll run and count as defeated. They should have something like this for fear and illusion spells.