So yeah, long story short, the problem goes away if you long rest a few times early.