Personally I think there should be a fatigue system. Time based, but also healing based. Time passes and fatigue increases. Healing also increases a character's fatigue. Resting resets fatigue. If you are out of short rests you would need a long rest to reset. Otherwise you are travelling exhausted with lower move speed and stats.
