Thinking about things, the one difficulty I feel like this creates is that it might be hard to justify those camp followers you can get; people who aren't part of your party and won't come out adventuring with you but tag along all the same. Figuring out how they could work would be awkward. But doable with the right amount of mage-babble handwaving.

Also how often are people taking long rests in camp? I am really hesitant and have only taken 4 long rests after playing for 18 hours or so, and I want to know if I've been dumb for doing that.