The ogres do warn us that they'll attack everyone around if we call them, so basically anyone not currently in your party is fair game. So that will be why they're fighting people at your camp who aren't actually in your party, but I agree it's pretty weird they come with you.
It definitely sounds worth reporting as a bug to Larian if you haven't already (using the Report a Bug link up at the top of the launcher). They might confirm whether it's by design or not, and if not it could get added to the backlog for fixing at some point.
