I had an issue with my party not fast travelling with me from camp to a destination when I used the portal to do so. My work around was that if I use the bedrolls to exit the camp, then my part came with me. Not sure if this is just a quick way to travel by yourself without having to individually tell your party members to chill. Enjoying the game so far.
