Entered camp for the night and the party with the refugees with 2 of my characters level 5 and 2 of them level 4 (due to inspiration xp). In the morning, when we spoke to Halsin, we got enough xp to put the other 2 over to level 5. Instead, they remained level 4 with over 6k xp to go to level 5, putting them a full level behind the other half of the party.

Luckily this bug can be worked around with a little gold and time by letting Withers respec the affected characters. They level up to 5 correctly after the respec.