Does it matter if you are trying to connect to another lobby or someone is trying to connect to yours?

Have you tried connecting via IP address? Unfortunately, even if that works better it wouldn't help for the ladder skirmish.