Series X. Same here. My friends keep timing out from my lobbies. Started happening in the late December update and when we reached Act 2. Not sure which was the cause.
Also experiencing the frame drops after 2-3 hours of gameplay or after having saved the game many times or even loading a save many times. Again, not sure the cause, but seemingly all connected.
