Yes, please for the love of god remove this mechanic. It's annoying, immersion breaking, and causes numerous issues. NPCs will follow you around and bark nonsense at your summons even when it makes zero sense and it makes me want to stop using summons altogether.
Super annoying, please please remove this "feature".