I feel as if the solution to this is fairly simple. When teleport is used on an actor, a status affect is placed on them known as "rift sickness" or something of the like. The status would grant immunity or high resistance against teleportation magic for (insert number here) turns. Or perhaps even state that it can only be used on an actor once per combat.
