Needs Attention No connection to Internet using version 1.6.8.0

[DQ]

I am not sure if this is helpful. But in my work I deal with a lot of tunneling. So things like MSS and MTU can often be an issue and I can have hops where we have tunnels under tunnels.

I see in your tcpdump that on the syn going out to Redfox your MSS is 1460. That is standard for an ethernet 1500 mtu. Now I see in that dump and the talk here that http (port 80) is being used. So I assume there are no certificates being exchanged, but if there are those tend to be the canary in the coal mine because that certificate exchange will use full sized packets. So if there is a tunnel somewhere that drops the MTU and nothing figures that out (Windows SHOULD through MTU discovery but in Win7 that detection might be weak and I am not sure when it was added to windows) much of your packets would pass but your cert exchange would get dropped as those are typically set with the do not fragment flag so the the upstream device that gets the oversized packet just drops it. And that fools you into believing everything is OK because you can ping all day long no problem and pass other traffic but the cert exchange gets dropped and HTTPS or whatever is using TLS fails.

Just a thought.

 

[Pete]

I briefly thought about MTU issues, too.
But the strange thing seems to be here, that everything is fine after the initial 15 seconds and that would not support that suspicion.
Let's wait, what ocd makes of the testversion with increased timeout.

 

[ocd]

Nicely done. Great intuition about the 15-second startup delay moving from annoyance to show-stopper.

With the increased timeout, 1.6.8.1 works. There's no real change in observed traffic, but apparently that increased timeout is enough to let things work after the 15-second delay.

14:09:20.899769 IP 172.16.185.130.55785 > 172.16.185.2.domain: 19836+ A? wpad.localdomain. (34)
14:09:21.902978 IP 172.16.185.130.55785 > 172.16.185.2.domain: 19836+ A? wpad.localdomain. (34)
14:09:23.914982 IP 172.16.185.130.55785 > 172.16.185.2.domain: 19836+ A? wpad.localdomain. (34)

14:09:27.924923 IP6 fe80::1c79:6a12:xxxx:xxxx.50282 > ff02::1:3.llmnr: UDP, length 22
14:09:27.925267 IP 172.16.185.130.61833 > 224.0.0.252.llmnr: UDP, length 22
14:09:28.032953 IP6 fe80::1c79:6a12:xxxx:xxxx.50282 > ff02::1:3.llmnr: UDP, length 22
14:09:28.033118 IP 172.16.185.130.61833 > 224.0.0.252.llmnr: UDP, length 22
14:09:28.235539 IP 172.16.185.130.netbios-ns > 172.16.185.2.netbios-ns: UDP, length 50
14:09:29.749667 IP 172.16.185.130.netbios-ns > 172.16.185.2.netbios-ns: UDP, length 50
14:09:31.262873 IP 172.16.185.130.netbios-ns > 172.16.185.2.netbios-ns: UDP, length 50
14:09:32.806553 IP 172.16.185.130.netbios-ns > 172.16.185.255.netbios-ns: UDP, length 50
14:09:33.570584 IP 172.16.185.130.netbios-ns > 172.16.185.255.netbios-ns: UDP, length 50
14:09:34.335008 IP 172.16.185.130.netbios-ns > 172.16.185.255.netbios-ns: UDP, length 50

14:09:35.104672 IP 172.16.185.130.50492 > 172.16.185.2.domain: 53885+ A? opd.redfox.bz. (31)
14:09:35.106841 IP 172.16.185.2.domain > 172.16.185.130.50492: 53885 1/0/0 A 185.246.128.167 (47)
14:09:35.109383 IP 172.16.185.130.49309 > se.redfox.bz.http: Flags [S], seq 2428477363, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
...
14:09:35.458957 IP 172.16.185.130.49309 > se.redfox.bz.http: Flags [.], ack 4909, win 64240, length 0
14:09:35.865041 IP 172.16.185.130.59835 > 172.16.185.2.domain: 43246+ A? wpad.localdomain. (34)
14:09:35.865345 IP 172.16.185.130.52332 > 172.16.185.2.domain: 25518+ A? wpad.localdomain. (34)
14:09:36.845378 IP 172.16.185.130.55527 > 172.16.185.2.domain: 39573+ A? (provider) (32)
14:09:36.873164 IP 172.16.185.2.domain > 172.16.185.130.55527: 39573 3/0/0 CNAME (provider)
14:09:36.875600 IP 172.16.185.130.52332 > 172.16.185.2.domain: 25518+ A? wpad.localdomain. (34)
14:09:36.875707 IP 172.16.185.130.59835 > 172.16.185.2.domain: 43246+ A? wpad.localdomain. (34)
...normal traffic...

But your comments about the mysterious delay intrigued me, and I noticed that the "wpad" queries always appeared (starting with DNS, then LLMNR, then NetBIOS) in that 15-second interval.

So I turned off WPAD (Control Panel -> Internet Options -> Connections -> LAN settings, deselect "Automatically detect settings") and no more 15-second delay!

And, for an added bonus, now 1.6.8.0 works.

Thank you for your help chasing this down!

 

[Pete]

Well, that's a happy ending
Thanks for the input.

Now, I'll have to think this through a bit.
Because wpad really is default on in Windows and for some reason, so far it only seems to be a problem on your end.
But that doesn't mean, nobody will run into this later on.

 

[ocd]

I wonder if it's my network's lack of NX response to its wpad.localdomain. queries? The lack of response to the UDP queries seems less remarkable.

 

[DQ]

Good find but that seems real odd to me. If nothing answers WPAD I personally fail to see how it's mucking up the works. But I will add that I have run into some really squirrely stuff with old Windows versions. They have mechanisms in them that have changed greatly over the years and they work but you end up with weird things sometimes that assume happen to protocol/stack differences.

I once had a situation with traffic going over multiple tunnels. A modern machine (win10/11) would fail the access. But an old WindowsXP laptop worked. Now the whole situation was a bug I found in the OS of a device where it was not properly calculating it's return MSS over it's own tunnel and would drop it's own traffic as it moved through the processes on the device. But the XP laptop for some reason I could not explain (I had no access to it) lowered it's MSS initially (like it was manually set) and worked fine. This gave the artificial view that the laptop was at fault.

Weird stuff.

 

[DQ]

I wonder if it's my network's lack of NX response to its wpad.localdomain. queries? The lack of response to the UDP queries seems less remarkable.

Nothing should answer it really. It's going to look for something from DHCP or DNS so it can get a config file. But that's all ancient stuff that is rarely if ever used anymore.

 

[cartman0208]

So I turned off WPAD (Control Panel -> Internet Options -> Connections -> LAN settings, deselect "Automatically detect settings") and no more 15-second delay!

Isn't that basically what I suggested in #9 ?

 

[Pete]

Isn't that basically what I suggested in #9 ?

It actually is, but it probably got drowned in all the chatter. Or - actually "autodetect" is commonly interpreted as "proxy off". Which it normally equates to. So nobody thought, there is a need to change anything.

Good find but that seems real odd to me. If nothing answers WPAD I personally fail to see how it's mucking up the works.

I do actually have a good picture of it in my head.
The Windows default setting is "auto detect". That's in a sense the smartest default setting to have, as it will work in more environments, than a default of "no proxy".

Now, Windows starts up and sends a couple of wpad discovery requests, gets no answer and all is good. Applications using WinHTTP will get launched and WinHTTP is already all set with whatever information wpad came up with.

AnyStream used WinHTTP for "historical" reasons, but due to the VSO debacle, switched to Qt's http implementation. Qt doesn't base on WinHTTP, because not all OSes have an equivalent to WinHTTP, so for portability has its own stuff.

Qt starts up, uses the Windows settings, sees "oh, I have to do a wpad discovery first". So it has to block network requests until it hat its wpad information.
So all network traffic will stall until the wpad matter is resolved, if there is no device in the network, that will answer.

There is a workaround, though, I can let AnyStream, instead of just "use OS settings", explicitly query the effective proxy settings (already resolved by the OS) and take over those.

So it's not a Windows problem, it's really a Qt problem. Would happen the same way on all OS's given the same scenario.

 

[DQ]

It actually is, but it probably got drowned in all the chatter. Or - actually "autodetect" is commonly interpreted as "proxy off". Which it normally equates to. So nobody thought, there is a need to change anything.


I do actually have a good picture of it in my head.
The Windows default setting is "auto detect". That's in a sense the smartest default setting to have, as it will work in more environments, than a default of "no proxy".

Now, Windows starts up and sends a couple of wpad discovery requests, gets no answer and all is good. Applications using WinHTTP will get launched and WinHTTP is already all set with whatever information wpad came up with.

AnyStream used WinHTTP for "historical" reasons, but due to the VSO debacle, switched to Qt's http implementation. Qt doesn't base on WinHTTP, because not all OSes have an equivalent to WinHTTP, so for portability has its own stuff.

Qt starts up, uses the Windows settings, sees "oh, I have to do a wpad discovery first". So it has to block network requests until it hat its wpad information.
So all network traffic will stall until the wpad matter is resolved, if there is no device in the network, that will answer.

There is a workaround, though, I can let AnyStream, instead of just "use OS settings", explicitly query the effective proxy settings (already resolved by the OS) and take over those.

So it's not a Windows problem, it's really a Qt problem. Would happen the same way on all OS's given the same scenario.

Thanks for explaining that Pete. Much appreciated. I see what you mean about Qt. I am assuming what you are eluding to is that it either should not halt traffic while performing queries or the wait timer should be far shorter.

In my opinion, if it's possible to have Qt not participate in WPAD that would be the best (one less mechanism to foul up the works). Or if there was a proxy option to turn it on or off but leave it default off. I have no idea if either thing is possible. But my take on it is, that is a very old protocol. In almost 3 decades of networking I have not seen it used once. This is not to say no one uses it but I would imagine any such use case at this point would involve a corporate network. I doubt someone is going to config their DNS or DHCP settings for WPAD usage or host a config file for it locally. Maybe there are some ISP use cases, not sure, not seen that but I only live in 1 country.

That's just my 2 cents on that. I am not a developer, just an old network dude.

 

[Pete]

Thanks for explaining that Pete. Much appreciated. I see what you mean about Qt. I am assuming what you are eluding to is that it either should not halt traffic while performing queries or the wait timer should be far shorter.

Yes.
The fact, that Qt holds back all traffic, until wpad is "resolved" is the right thing to do - it can't just start pumping packets before it knows, how it is supposed to do that properly.
But 15 seconds is a looong timeout for a protocol, that should normally be dealt with in a LAN. If you don't have your wpad answer within 1 or 2 seconds, there won't be one.

I decided to throw a message, if the first connection takes overly long AND wpad is active - AnyStream will ask whether it should disable wpad.

 

[DQ]

Yes.
The fact, that Qt holds back all traffic, until wpad is "resolved" is the right thing to do - it can't just start pumping packets before it knows, how it is supposed to do that properly.
But 15 seconds is a looong timeout for a protocol, that should normally be dealt with in a LAN. If you don't have your wpad answer within 1 or 2 seconds, there won't be one.

I decided to throw a message, if the first connection takes overly long AND wpad is active - AnyStream will ask whether it should disable wpad.

Thanks Pete. I think that's a great solution. I agree (not that you need my agreement) that 15 seconds for a protocol timeout is obnoxious. I work on cellular networks which are typically very slow and latent compared to a LAN or even a typical WAN. My timeouts for things are typically 3-5 secs. Sometimes on horrible connections I will go to 10. So 15 seconds on a LAN is just dysfunctional in my view to the point I believe they made a mistake unintentionally.

I also think 1-2 seconds for a wait period is perfect. From what I understand modern WPAD only uses a DNS query at this point or that is the primary method of discovery now.

I also performed some real world testing. I turned WPAD back on my PC. I opened Firefox. It ran 2 DNS queries for WPAD in rapid succession but in my case it did not have to timeout as my local DNS server (PiHole) sent negative responses. I tried this with my Windows DNS box and it also sent a negative response. Maybe the case is if you are using ISP assigned DNS there is no negative response and as such the timeout must occur. In that case, and maybe WPAD has it's own timers, but on a pure DNS basis the default Windows DNS timeout (when 2 servers are configured) is 9 seconds if nothing answers.

This is a interesting issue all around. This has convinced me to disable WPAD on all my machines.