I ran into something similar, if you split to a single character you should be able to leave then have the rest of your party teleport to you.