Honestly just move it to a party of six and maybe say something about how more than that would be too conspicuous, thats it. Also have it that a recruited companion who is not active can still go forward when its relevant, like how it does not make any sense for Laezel to hang back at camp when you encounter the gith (yet everyone still knows in real time and judges what you do.)
