I've never had this kind of issues. Unless I split the party, the whole party always follows whichever party member is in my direct control. Summons have their own slightly different rules.