Same issue but singleplayer here with custom party. Ive noticed that when I put a character in/out character in Withers Wardrobe theres a chance that they might stop following you (sometime it'll happen, sometimes it will be fine). I dont know what exactly triggers it but it seems to be an issue with the wardrobe.
Like you, Ive tried everything I could think but nothing worked. The only workaround I found is reload an earlier save that doesnt have this bug.
