If you can survive in Broken Valley, then feel free to do so. You might end up a little over-leveled when you return to the fjords, though, depending on how much you've done already.