You can leave and return as much as you wish, until you turn in the quest and tell Louis the location. After that you can return, but the bandits inside will be hostile. The ones on the outside stay hostile regardless (at least I think they do; I cleared them out before entering).
You can even side with the merchants in the robbery quest, kill the bandits, and return to camp and report back on the outcome (interesting dialog options for that).
