You could do this. But there isn't any good reason to do so. There is at least one unavoidable fight at the end of the act, and while you probably can make it to most places on the map while avoiding combat, sooner or later you'll need to fight.
level 3 voidvoken dweller ambush *shudders*
