Did you ask someone in your party to stay at camp first?

The other's won't let you switch then into your party without free space.