That can also happen if the pyramid is in a container (sent to the Homestead, etc). Using it on a different map will still teleport you to the actual map's coordinates.