How can you get there too late? i know
I read some time ago that If you go there and leave again, long rest, and return -- the people will have burned to death. That's a way to miss out on some quests. So, I made sure to avoid it. I haven't read anything about too many long rests endangering them if the area remains unloaded, though. I don't know how close you have to get, so I just do it early. Free exp and no enemies, perfect for level 3.
The Zhentarim trader also has some really good items. If you steal the iron flask, then torture the guy to get back into their good graces, they trust you enough for access lol. I picked up some glove that gives advantage on all sleight of hand checks. Insane. I usually murderhobo them, but it's worth waiting out multiple shops resets before that. Who knows what else they have.