The only thing this might impact is if you take a long rest in an unsafe area and the camp gets attacked, you would then have more followers under your control than normal, but I don't really think that's such a huge issue.
That could be dealt with by having the camp attack happen before you get to the actual 'safe' camp. After combat is resolved the safe camp with all members opens up.
