You need to clear the smallish area he teleports you to, but after that you get your teleporter stones and are able to return and complete previous quests, etc or do new ones. There is a one-way trip, but not until you have completed the new quests gotten in the area he teleports you to, and ring a certain gong (you can ring it before then without adverse effect).
