Hrm, we have both been in-game when we sent invites to each other. I will try your method of having one player out of the game when they receive the invite.
Otherwise our set up is the same. My xb1 is wired, while hers is wireless over 802.11n.
I'll post back with results when I can go try it out