You are suppose to loose the teleporter pyramids at the start of act 4 as part of the plot. They should have had the imp historian show up to retrieve them, or something, to make this known.