If the character that you are not controlling follows you, then they are linked together (which is the default configuration). You can click on a portrait and drag it a bit to the right to unlink it, and drag it near another portrait to link it.