Imo, the number of rests (so unlimited atm) is not really the problem. Rather from WHERE I can go to camp. I shouldn't be able to go to camp from the inside of the selune temple, after killing Raghzlin, when there are still goblins about.