To expand on that further, its just a little quirk, the return/exit button returns you from the last spot you used the jewel to teleport to the tower, and since the jewel is brand new there is no "last place" for you to teleport back to, so it fails.
