that's actually part of the gameplay and totally on purpose
At least the idea of Cave of the Shipwreckers was to ensure that the characters were individually selected from different places and reunited. And these manipulations break it down.
I also met a few more places where you can abuse it and as a whole it even spoils the atmosphere of the game, the world seems not so holistic and when you get into a trap you always keep in mind "I can press the teleport button at any time, it's okay" and you no longer feel trapped.
