This is an annoying bug. I saw it in early-access and it's still with us after launch.
It's becoming a regular chore now, to manually reorder the party back into the default order, if i go to camp or fast travel.
My default party order is to have my custom character as F1, my fighter as F2, rogue as F3 and caster as F4.
The BUG is that when I go to camp, or open the map and use an Ancient Sigil Circle to fast travel, then F3 and F2 will swap always places. Sometimes other characters swap places, but I have not had that happen enough to document how or when this happens.
For example. I'm in the wilderness, not under attack, party is grouped up in my default order above. I decide to go to camp, or fast travel, and when I arrive at the destination party members in slots F2 & F3 will swap position in the group.
Every time i fast travel using the map, characters swap places. If I go to camp, they swap places, but if I reoder them to default in camp, they will not swap places when I leave camp.
So to recap, fast traveling using the sigils always swaps F2 & F3. Only going to camp swaps them, leaving camp the order does not change; providing I swapped them back to default order while in camp.