Yes. When you are beat down and used up all your spells, the exact thing you need is a tough battle suddenly hitting you before you can rest. Nothing makes a game playable like assured death.

Random encounters belong in the overall world, not at camp which you don't access until you absolutely need it.