I'm not sure how the Xbox servers are set up. It is possible if you are having trouble with some friends and not others, that the problem is just on the other end when it doesn't work.