Just curiosity on my part, but
1. How often have you long rested?
2. Have you gotten close to the location before?

One of my games is buggy as all hell, but that location wasn't burned down on day 2