I don't mean to be rude, but you're allowed to speak to each other. Relay the gist of the conversation, this way it also gets more social. I really like doing it like this, anyways, and even though I might have missed a bit of the story, I have had a blast.