You definitely can, which on a side note implies that nothing that happens in those locations matters THAT much to the overall narrative.
You'd be missing out on a bunch of experience and loot, as well a number of companion interactions, so your relationships with companions would be very neutral.
The only reason you'd want to is that you're dead set on remaining neutral in others' conflicts, or you're speed-running