I cleared a fair chunk of Broken Valley before I got around to visiting Lovis. I took out a few of the level 11 black ring soldiers outside Maxos temple when I was level 9, but they were a little too tough in groups of more than 2, so I avoided them and headed further west.
If you save periodically, or before entering new buildings / caves, etc then you should be fine even if you start running into trouble.
