I experienced this same issue with me as host and a remote player. We were able to clear this up by restarting the game and inviting my buddy back in. We always had this issue only when my buddy was in game and we reloaded. If we happened to reload and got into this state, I had to shut the game completely down and sometimes restart my machine to get the game going again.
