This is actually a part of the plot line, when you enter act IV, you loose the teleporter stones.
There really should be a cut scene warning you about this, but there isn't.

I used to be snow white, but I drifted... Check out my Beyond Divinity fansite