It is poorly designed. And it's sad because banter is cool (this is a universal truth), and I think quite few people will be happy to miss it.
It has at least three solutions (probably more) :
- Reduce the radius of action of the camera. Of course, that means finding other ways to do medium-fast travel : I don't want to spend half the game clicking on the ground.
- Don't mute in-party banter. They should not be treated the same way as banter between out-of-party characters.
- When banter should trigger, delay it until the camera is back near the characters (as Verte described).
These solutions are not mutually exclusive.