Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Windows node traffic interruptions #9219

Closed
Argannor opened this issue Sep 11, 2024 · 2 comments
Closed

Windows node traffic interruptions #9219

Argannor opened this issue Sep 11, 2024 · 2 comments
Labels
area/windows

Comments

@Argannor
Copy link

Argannor commented Sep 11, 2024 •
edited
Loading

We're running Calico on an AWS EKS cluster with both Windows and Linux nodes. When running long running downloads via HTTP(s), we observe connection timeouts, which are caused by missing ACK packages from the client. The client is running on windows nodes inside the cluster.

It does not matter, whether the server receiving the requests is outside of the cluster or on Linux nodes, the behavior is the same.

Any help on this one is greatly appreciated, I'm uncertain whether this problem is caused by Calico or something else.

Expected Behavior

Downloads of files should succeed and TCP packages should be ACKnowledged.

Current Behavior

As outlined above, when downloading a larger file i.e. 200MB (the client does not matter, we tested curl.exe, C# code, or Invoke-WebRequest), the download will first start (although slow) and after a while (observed times between 30s and 3minutes) the progress will stall. Ultimately the connection will be forcibly closed by the server as the client does not read the rest of the response body.

When capturing the traffic with netsh trace start capture=yes and netsh trace stop and analyzing the resulting trace with Wireshark, we can observe the lack of ACK packages at some point.:

353785	100.998070	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188647958 Ack=327 Win=62080 Len=1410
353786	100.998070	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188649368 Ack=327 Win=62080 Len=1276
353787	100.998071	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188650644 Ack=327 Win=62080 Len=1410
353788	100.998072	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188652054 Ack=327 Win=62080 Len=1410
353789	100.998073	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188653464 Ack=327 Win=62080 Len=1276
353790	100.998073	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188654740 Ack=327 Win=62080 Len=1410
353791	100.998074	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188656150 Ack=327 Win=62080 Len=1410
353792	100.998075	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188657560 Ack=327 Win=62080 Len=1276
353793	100.998233	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188646548 Ack=327 Win=62080 Len=1410
353794	100.998234	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188647958 Ack=327 Win=62080 Len=1410
353795	100.998235	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188649368 Ack=327 Win=62080 Len=1276
353796	100.998235	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188650644 Ack=327 Win=62080 Len=1410
353797	100.998236	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188652054 Ack=327 Win=62080 Len=1410
353798	100.998236	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188653464 Ack=327 Win=62080 Len=1276
353799	100.998237	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188654740 Ack=327 Win=62080 Len=1410
353800	100.998237	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188656150 Ack=327 Win=62080 Len=1410
353801	100.998238	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188657560 Ack=327 Win=62080 Len=1276
353802	100.998266	192.168.35.164	172.20.121.29	TCP	54	55791 → 8080 [ACK] Seq=327 Ack=188658836 Win=262144 Len=0
353803	100.998282	192.168.35.164	192.168.114.23	TCP	104	55791 → 8080 [ACK] Seq=327 Ack=188658836 Win=262144 Len=0
353804	100.998296	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188658836 Ack=327 Win=62080 Len=1410
353805	100.998297	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188660246 Ack=327 Win=62080 Len=1410
353806	100.998298	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188661656 Ack=327 Win=62080 Len=1276
353807	100.998380	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188658836 Ack=327 Win=62080 Len=1410
353808	100.998381	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188660246 Ack=327 Win=62080 Len=1410
353809	100.998382	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188661656 Ack=327 Win=62080 Len=1276
353810	100.998405	192.168.35.164	172.20.121.29	TCP	54	55791 → 8080 [ACK] Seq=327 Ack=188662932 Win=262144 Len=0
353811	100.998419	192.168.35.164	192.168.114.23	TCP	104	55791 → 8080 [ACK] Seq=327 Ack=188662932 Win=262144 Len=0
353813	101.002013	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188662932 Ack=327 Win=62080 Len=1410
353814	101.002015	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188664342 Ack=327 Win=62080 Len=1410
353815	101.002015	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188665752 Ack=327 Win=62080 Len=1276
353816	101.002017	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188667028 Ack=327 Win=62080 Len=1410
353817	101.002017	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188668438 Ack=327 Win=62080 Len=1410
353818	101.002018	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188669848 Ack=327 Win=62080 Len=1276
353819	101.002019	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188671124 Ack=327 Win=62080 Len=1410
353820	101.002019	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188672534 Ack=327 Win=62080 Len=1410
353821	101.002020	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188673944 Ack=327 Win=62080 Len=1276
353822	101.002021	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188675220 Ack=327 Win=62080 Len=1410
353823	101.002021	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188676630 Ack=327 Win=62080 Len=1410
353824	101.002022	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188678040 Ack=327 Win=62080 Len=1276
353826	101.002287	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188662932 Ack=327 Win=62080 Len=1410
353827	101.002288	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188664342 Ack=327 Win=62080 Len=1410
353828	101.002289	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188665752 Ack=327 Win=62080 Len=1276
353829	101.002289	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188667028 Ack=327 Win=62080 Len=1410
353830	101.002290	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188668438 Ack=327 Win=62080 Len=1410
353831	101.002290	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188669848 Ack=327 Win=62080 Len=1276
353832	101.002291	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188671124 Ack=327 Win=62080 Len=1410
353833	101.002291	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188672534 Ack=327 Win=62080 Len=1410
353834	101.002292	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188673944 Ack=327 Win=62080 Len=1276
353835	101.002292	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188675220 Ack=327 Win=62080 Len=1410
353836	101.002293	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188676630 Ack=327 Win=62080 Len=1410
353837	101.002293	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188678040 Ack=327 Win=62080 Len=1276
353838	101.002334	192.168.35.164	172.20.121.29	TCP	54	55791 → 8080 [ACK] Seq=327 Ack=188679316 Win=262144 Len=0
353839	101.002353	192.168.35.164	192.168.114.23	TCP	104	55791 → 8080 [ACK] Seq=327 Ack=188679316 Win=262144 Len=0
353870	101.064408	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
353871	101.064410	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188680726 Ack=327 Win=62080 Len=1410
353872	101.064411	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188682136 Ack=327 Win=62080 Len=1276
353873	101.064411	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188683412 Ack=327 Win=62080 Len=1410
353874	101.064412	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188684822 Ack=327 Win=62080 Len=1410
353875	101.064571	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
353876	101.064572	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188680726 Ack=327 Win=62080 Len=1410
353877	101.064572	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188682136 Ack=327 Win=62080 Len=1276
353878	101.064573	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188683412 Ack=327 Win=62080 Len=1410
353879	101.064574	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188684822 Ack=327 Win=62080 Len=1410
353880	101.064753	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188686232 Ack=327 Win=62080 Len=1276
353881	101.064753	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188687508 Ack=327 Win=62080 Len=1410
353882	101.064754	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188688918 Ack=327 Win=62080 Len=1410
353883	101.064755	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188690328 Ack=327 Win=62080 Len=1276
353884	101.064755	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188691604 Ack=327 Win=62080 Len=1410
353885	101.064756	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188693014 Ack=327 Win=62080 Len=1410
353886	101.064757	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188694424 Ack=327 Win=62080 Len=1276
353887	101.064757	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188695700 Ack=327 Win=62080 Len=1410
353888	101.064758	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188697110 Ack=327 Win=62080 Len=1410
353889	101.064759	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188698520 Ack=327 Win=62080 Len=1276
353890	101.064759	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188699796 Ack=327 Win=62080 Len=1410
353891	101.064760	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188701206 Ack=327 Win=62080 Len=1410
353892	101.064761	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188702616 Ack=327 Win=62080 Len=1276
353895	101.064763	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188703892 Ack=327 Win=62080 Len=1410
353896	101.065192	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188686232 Ack=327 Win=62080 Len=1276
353897	101.065194	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188687508 Ack=327 Win=62080 Len=1410
353898	101.065194	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188688918 Ack=327 Win=62080 Len=1410
353899	101.065195	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188690328 Ack=327 Win=62080 Len=1276
353900	101.065196	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188691604 Ack=327 Win=62080 Len=1410
353901	101.065196	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188693014 Ack=327 Win=62080 Len=1410
353902	101.065197	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188694424 Ack=327 Win=62080 Len=1276
353903	101.065197	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188695700 Ack=327 Win=62080 Len=1410
353904	101.065198	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188697110 Ack=327 Win=62080 Len=1410
353905	101.065199	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188698520 Ack=327 Win=62080 Len=1276
353906	101.065199	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188699796 Ack=327 Win=62080 Len=1410
353907	101.065200	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188701206 Ack=327 Win=62080 Len=1410
353908	101.065201	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188702616 Ack=327 Win=62080 Len=1276
353911	101.065458	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188703892 Ack=327 Win=62080 Len=1410
353912	101.065551	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188705302 Ack=327 Win=62080 Len=1410
353913	101.065552	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188706712 Ack=327 Win=62080 Len=1276
353914	101.065553	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188707988 Ack=327 Win=62080 Len=1410
353915	101.065554	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188709398 Ack=327 Win=62080 Len=1410
353916	101.065554	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188710808 Ack=327 Win=62080 Len=1276
353917	101.065556	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188712084 Ack=327 Win=62080 Len=1410
353918	101.065557	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188713494 Ack=327 Win=62080 Len=1410
353919	101.065558	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188714904 Ack=327 Win=62080 Len=1276
353920	101.065559	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188716180 Ack=327 Win=62080 Len=1410
353921	101.065560	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188717590 Ack=327 Win=62080 Len=1410
353922	101.065560	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188719000 Ack=327 Win=62080 Len=1276
353923	101.065561	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188720276 Ack=327 Win=62080 Len=1410
353924	101.065562	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188721686 Ack=327 Win=62080 Len=1410
353925	101.065563	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188723096 Ack=327 Win=62080 Len=1276
353926	101.065564	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188724372 Ack=327 Win=62080 Len=1410
353927	101.065565	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188725782 Ack=327 Win=62080 Len=1410
353928	101.065565	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188727192 Ack=327 Win=62080 Len=1276
353929	101.065566	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188728468 Ack=327 Win=62080 Len=1410
353930	101.065567	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188729878 Ack=327 Win=62080 Len=1410
353931	101.065568	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188731288 Ack=327 Win=62080 Len=1276
353932	101.065569	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188732564 Ack=327 Win=62080 Len=1410
353933	101.065570	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188733974 Ack=327 Win=62080 Len=1410
353934	101.065571	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188735384 Ack=327 Win=62080 Len=1276
353935	101.065572	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188736660 Ack=327 Win=62080 Len=1410
353936	101.065573	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188738070 Ack=327 Win=62080 Len=1410
353937	101.065574	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188739480 Ack=327 Win=62080 Len=1276
353938	101.065574	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188740756 Ack=327 Win=62080 Len=1410
353939	101.065575	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188742166 Ack=327 Win=62080 Len=1410
353940	101.065576	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188743576 Ack=327 Win=62080 Len=1276
353941	101.065577	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188744852 Ack=327 Win=62080 Len=1410
353942	101.065577	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188746262 Ack=327 Win=62080 Len=1410
353943	101.065578	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188747672 Ack=327 Win=62080 Len=1276
353944	101.065579	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188748948 Ack=327 Win=62080 Len=1410
353945	101.065580	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188750358 Ack=327 Win=62080 Len=1410
353946	101.065581	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188751768 Ack=327 Win=62080 Len=1276
353947	101.065582	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188753044 Ack=327 Win=62080 Len=1410
353948	101.065583	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188754454 Ack=327 Win=62080 Len=1410
353949	101.065584	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188755864 Ack=327 Win=62080 Len=1276
353950	101.065585	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188757140 Ack=327 Win=62080 Len=1410
353951	101.065586	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188758550 Ack=327 Win=62080 Len=1410
353952	101.065587	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188759960 Ack=327 Win=62080 Len=1276
353953	101.066300	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188705302 Ack=327 Win=62080 Len=1410
353954	101.066301	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188706712 Ack=327 Win=62080 Len=1276
353955	101.066302	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188707988 Ack=327 Win=62080 Len=1410
353956	101.066302	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188709398 Ack=327 Win=62080 Len=1410
353957	101.066303	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188710808 Ack=327 Win=62080 Len=1276
353958	101.066303	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188712084 Ack=327 Win=62080 Len=1410
353959	101.066304	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188713494 Ack=327 Win=62080 Len=1410
353960	101.066305	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188714904 Ack=327 Win=62080 Len=1276
353961	101.066305	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188716180 Ack=327 Win=62080 Len=1410
353962	101.066306	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188717590 Ack=327 Win=62080 Len=1410
353963	101.066306	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188719000 Ack=327 Win=62080 Len=1276
353964	101.066307	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188720276 Ack=327 Win=62080 Len=1410
353965	101.066307	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188721686 Ack=327 Win=62080 Len=1410
353966	101.066308	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188723096 Ack=327 Win=62080 Len=1276
353967	101.066308	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188724372 Ack=327 Win=62080 Len=1410
353968	101.066309	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188725782 Ack=327 Win=62080 Len=1410
353969	101.066309	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188727192 Ack=327 Win=62080 Len=1276
353970	101.066310	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188728468 Ack=327 Win=62080 Len=1410
353971	101.066310	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188729878 Ack=327 Win=62080 Len=1410
353972	101.066311	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188731288 Ack=327 Win=62080 Len=1276
353973	101.066311	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188732564 Ack=327 Win=62080 Len=1410
353974	101.066312	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188733974 Ack=327 Win=62080 Len=1410
353975	101.066312	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188735384 Ack=327 Win=62080 Len=1276
353976	101.066313	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188736660 Ack=327 Win=62080 Len=1410
353977	101.066313	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188738070 Ack=327 Win=62080 Len=1410
353978	101.066314	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188739480 Ack=327 Win=62080 Len=1276
353979	101.066314	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188740756 Ack=327 Win=62080 Len=1410
353980	101.066315	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188742166 Ack=327 Win=62080 Len=1410
353981	101.066315	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188743576 Ack=327 Win=62080 Len=1276
353982	101.066316	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188744852 Ack=327 Win=62080 Len=1410
353983	101.066316	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188746262 Ack=327 Win=62080 Len=1410
353984	101.066317	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188747672 Ack=327 Win=62080 Len=1276
353985	101.066317	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188748948 Ack=327 Win=62080 Len=1410
353986	101.066317	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188750358 Ack=327 Win=62080 Len=1410
353987	101.066318	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188751768 Ack=327 Win=62080 Len=1276
353988	101.066318	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188753044 Ack=327 Win=62080 Len=1410
353989	101.066319	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188754454 Ack=327 Win=62080 Len=1410
353990	101.066320	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188755864 Ack=327 Win=62080 Len=1276
353991	101.066320	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188757140 Ack=327 Win=62080 Len=1410
353992	101.066321	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188758550 Ack=327 Win=62080 Len=1410
353993	101.066321	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188759960 Ack=327 Win=62080 Len=1276
353994	101.066672	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188761236 Ack=327 Win=62080 Len=1410
353995	101.066673	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188762646 Ack=327 Win=62080 Len=1410
353996	101.066673	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188764056 Ack=327 Win=62080 Len=1276
353997	101.066674	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188765332 Ack=327 Win=62080 Len=1410
353998	101.066675	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188766742 Ack=327 Win=62080 Len=1410
353999	101.066676	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188768152 Ack=327 Win=62080 Len=1276
354000	101.066677	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188769428 Ack=327 Win=62080 Len=1410
354001	101.066678	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188770838 Ack=327 Win=62080 Len=1410
354002	101.066679	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188772248 Ack=327 Win=62080 Len=1276
354003	101.066680	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188773524 Ack=327 Win=62080 Len=1410
354004	101.066680	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188774934 Ack=327 Win=62080 Len=1410
354005	101.066681	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188776344 Ack=327 Win=62080 Len=1276
354006	101.066682	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188777620 Ack=327 Win=62080 Len=1410
354007	101.066683	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188779030 Ack=327 Win=62080 Len=1410
354008	101.066684	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188780440 Ack=327 Win=62080 Len=1276
354009	101.066685	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188781716 Ack=327 Win=62080 Len=1410
354010	101.066686	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188783126 Ack=327 Win=62080 Len=1410
354011	101.066687	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188784536 Ack=327 Win=62080 Len=1276
354012	101.066688	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188785812 Ack=327 Win=62080 Len=1410
354013	101.066688	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188787222 Ack=327 Win=62080 Len=1410
354014	101.066689	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188788632 Ack=327 Win=62080 Len=1276
354015	101.066690	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188789908 Ack=327 Win=62080 Len=1410
354016	101.066690	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188791318 Ack=327 Win=62080 Len=1410
354017	101.066691	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188792728 Ack=327 Win=62080 Len=1276
354018	101.067081	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188761236 Ack=327 Win=62080 Len=1410
354019	101.067082	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188762646 Ack=327 Win=62080 Len=1410
354020	101.067083	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188764056 Ack=327 Win=62080 Len=1276
354021	101.067084	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188765332 Ack=327 Win=62080 Len=1410
354022	101.067084	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188766742 Ack=327 Win=62080 Len=1410
354023	101.067085	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188768152 Ack=327 Win=62080 Len=1276
354024	101.067085	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188769428 Ack=327 Win=62080 Len=1410
354025	101.067086	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188770838 Ack=327 Win=62080 Len=1410
354026	101.067086	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188772248 Ack=327 Win=62080 Len=1276
354027	101.067087	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188773524 Ack=327 Win=62080 Len=1410
354028	101.067088	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188774934 Ack=327 Win=62080 Len=1410
354029	101.067088	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188776344 Ack=327 Win=62080 Len=1276
354030	101.067089	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188777620 Ack=327 Win=62080 Len=1410
354031	101.067089	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188779030 Ack=327 Win=62080 Len=1410
354032	101.067090	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188780440 Ack=327 Win=62080 Len=1276
354033	101.067090	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188781716 Ack=327 Win=62080 Len=1410
354034	101.067091	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188783126 Ack=327 Win=62080 Len=1410
354035	101.067092	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188784536 Ack=327 Win=62080 Len=1276
354036	101.067092	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188785812 Ack=327 Win=62080 Len=1410
354037	101.067093	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188787222 Ack=327 Win=62080 Len=1410
354038	101.067093	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188788632 Ack=327 Win=62080 Len=1276
354039	101.067094	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188789908 Ack=327 Win=62080 Len=1410
354040	101.067095	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188791318 Ack=327 Win=62080 Len=1410
354041	101.067095	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188792728 Ack=327 Win=62080 Len=1276
354042	101.070503	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188794004 Ack=327 Win=62080 Len=1410
354043	101.070504	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188795414 Ack=327 Win=62080 Len=1410
354044	101.070505	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188796824 Ack=327 Win=62080 Len=1276
354045	101.070642	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188794004 Ack=327 Win=62080 Len=1410
354046	101.070643	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188795414 Ack=327 Win=62080 Len=1410
354047	101.070643	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188796824 Ack=327 Win=62080 Len=1276
354048	101.070793	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188798100 Ack=327 Win=62080 Len=1410
354049	101.070794	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188799510 Ack=327 Win=62080 Len=1410
354050	101.070795	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188800920 Ack=327 Win=62080 Len=1276
354051	101.070796	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188802196 Ack=327 Win=62080 Len=1410
354052	101.070796	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188803606 Ack=327 Win=62080 Len=1410
354053	101.070797	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188805016 Ack=327 Win=62080 Len=1276
354054	101.070798	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188806292 Ack=327 Win=62080 Len=1410
354055	101.070798	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188807702 Ack=327 Win=62080 Len=1410
354056	101.070799	192.168.114.23	192.168.35.164	TCP	1380	8080 → 55791 [PSH, ACK] Seq=188809112 Ack=327 Win=62080 Len=1276
354057	101.070942	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188798100 Ack=327 Win=62080 Len=1410
354058	101.070943	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188799510 Ack=327 Win=62080 Len=1410
354059	101.070943	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188800920 Ack=327 Win=62080 Len=1276
354060	101.070944	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188802196 Ack=327 Win=62080 Len=1410
354061	101.070944	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188803606 Ack=327 Win=62080 Len=1410
354062	101.070945	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188805016 Ack=327 Win=62080 Len=1276
354063	101.070945	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188806292 Ack=327 Win=62080 Len=1410
354064	101.070946	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188807702 Ack=327 Win=62080 Len=1410
354065	101.070946	172.20.121.29	192.168.35.164	TCP	1330	8080 → 55791 [PSH, ACK] Seq=188809112 Ack=327 Win=62080 Len=1276
354066	101.084374	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188810388 Ack=327 Win=62080 Len=1410
354067	101.084376	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188811798 Ack=327 Win=62080 Len=1410
354068	101.084377	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188813208 Ack=327 Win=62080 Len=1410
354069	101.084378	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188814618 Ack=327 Win=62080 Len=1410
354070	101.084379	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188816028 Ack=327 Win=62080 Len=1410
354071	101.084380	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188817438 Ack=327 Win=62080 Len=1410
354072	101.084381	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188818848 Ack=327 Win=62080 Len=1410
354073	101.084381	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188820258 Ack=327 Win=62080 Len=1410
354074	101.084382	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188821668 Ack=327 Win=62080 Len=1410
354075	101.084383	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188823078 Ack=327 Win=62080 Len=1410
354076	101.084384	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188824488 Ack=327 Win=62080 Len=1410
354077	101.084384	192.168.114.23	192.168.35.164	TCP	978	8080 → 55791 [PSH, ACK] Seq=188825898 Ack=327 Win=62080 Len=874
354078	101.084661	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188810388 Ack=327 Win=62080 Len=1410
354079	101.084663	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188811798 Ack=327 Win=62080 Len=1410
354080	101.084664	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188813208 Ack=327 Win=62080 Len=1410
354081	101.084664	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188814618 Ack=327 Win=62080 Len=1410
354082	101.084665	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188816028 Ack=327 Win=62080 Len=1410
354083	101.084666	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188817438 Ack=327 Win=62080 Len=1410
354084	101.084666	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188818848 Ack=327 Win=62080 Len=1410
354085	101.084667	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188820258 Ack=327 Win=62080 Len=1410
354086	101.084667	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188821668 Ack=327 Win=62080 Len=1410
354087	101.084668	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188823078 Ack=327 Win=62080 Len=1410
354088	101.084668	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188824488 Ack=327 Win=62080 Len=1410
354089	101.084669	172.20.121.29	192.168.35.164	TCP	928	8080 → 55791 [PSH, ACK] Seq=188825898 Ack=327 Win=62080 Len=874
354102	101.090920	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188826772 Ack=327 Win=62080 Len=1410
354103	101.090923	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188828182 Ack=327 Win=62080 Len=1410
354104	101.090923	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188829592 Ack=327 Win=62080 Len=1410
354105	101.090924	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188831002 Ack=327 Win=62080 Len=1410
354106	101.090925	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188832412 Ack=327 Win=62080 Len=1410
354107	101.090926	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188833822 Ack=327 Win=62080 Len=1410
354108	101.090926	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188835232 Ack=327 Win=62080 Len=1410
354109	101.090927	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188836642 Ack=327 Win=62080 Len=1410
354110	101.090928	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188838052 Ack=327 Win=62080 Len=1410
354111	101.090929	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188839462 Ack=327 Win=62080 Len=1410
354112	101.090929	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188840872 Ack=327 Win=62080 Len=1410
354113	101.090930	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188842282 Ack=327 Win=62080 Len=1410
354114	101.090931	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188843692 Ack=327 Win=62080 Len=1410
354115	101.090932	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188845102 Ack=327 Win=62080 Len=1410
354116	101.090933	192.168.114.23	192.168.35.164	TCP	844	8080 → 55791 [PSH, ACK] Seq=188846512 Ack=327 Win=62080 Len=740
354117	101.091255	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188826772 Ack=327 Win=62080 Len=1410
354118	101.091256	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188828182 Ack=327 Win=62080 Len=1410
354119	101.091256	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188829592 Ack=327 Win=62080 Len=1410
354120	101.091257	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188831002 Ack=327 Win=62080 Len=1410
354121	101.091258	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188832412 Ack=327 Win=62080 Len=1410
354122	101.091259	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188833822 Ack=327 Win=62080 Len=1410
354123	101.091259	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188835232 Ack=327 Win=62080 Len=1410
354124	101.091260	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188836642 Ack=327 Win=62080 Len=1410
354125	101.091260	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188838052 Ack=327 Win=62080 Len=1410
354126	101.091261	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188839462 Ack=327 Win=62080 Len=1410
354127	101.091261	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188840872 Ack=327 Win=62080 Len=1410
354128	101.091262	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188842282 Ack=327 Win=62080 Len=1410
354129	101.091262	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188843692 Ack=327 Win=62080 Len=1410
354130	101.091263	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188845102 Ack=327 Win=62080 Len=1410
354131	101.091263	172.20.121.29	192.168.35.164	TCP	794	8080 → 55791 [PSH, ACK] Seq=188846512 Ack=327 Win=62080 Len=740
354132	101.095055	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188847252 Ack=327 Win=62080 Len=1410
354133	101.095057	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188848662 Ack=327 Win=62080 Len=1410
354134	101.095057	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188850072 Ack=327 Win=62080 Len=1410
354135	101.095197	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188847252 Ack=327 Win=62080 Len=1410
354136	101.095204	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188848662 Ack=327 Win=62080 Len=1410
354137	101.095205	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188850072 Ack=327 Win=62080 Len=1410
354138	101.095305	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188851482 Ack=327 Win=62080 Len=1410
354139	101.095306	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188852892 Ack=327 Win=62080 Len=1410
354140	101.095307	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188854302 Ack=327 Win=62080 Len=1410
354141	101.095308	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188855712 Ack=327 Win=62080 Len=1410
354142	101.095308	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188857122 Ack=327 Win=62080 Len=1410
354143	101.095309	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188858532 Ack=327 Win=62080 Len=1410
354144	101.095321	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188859942 Ack=327 Win=62080 Len=1410
354145	101.095322	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188861352 Ack=327 Win=62080 Len=1410
354146	101.095323	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188862762 Ack=327 Win=62080 Len=1410
354147	101.095323	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188864172 Ack=327 Win=62080 Len=1410
354148	101.095324	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188865582 Ack=327 Win=62080 Len=1410
354149	101.095325	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [ACK] Seq=188866992 Ack=327 Win=62080 Len=1410
354150	101.095326	192.168.114.23	192.168.35.164	TCP	1514	8080 → 55791 [PSH, ACK] Seq=188868402 Ack=327 Win=62080 Len=1410
354151	101.095550	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188851482 Ack=327 Win=62080 Len=1410
354152	101.095551	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188852892 Ack=327 Win=62080 Len=1410
354153	101.095552	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188854302 Ack=327 Win=62080 Len=1410
354154	101.095552	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188855712 Ack=327 Win=62080 Len=1410
354155	101.095553	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188857122 Ack=327 Win=62080 Len=1410
354156	101.095553	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188858532 Ack=327 Win=62080 Len=1410
354157	101.095554	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188859942 Ack=327 Win=62080 Len=1410
354158	101.095554	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188861352 Ack=327 Win=62080 Len=1410
354159	101.095555	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188862762 Ack=327 Win=62080 Len=1410
354160	101.095555	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188864172 Ack=327 Win=62080 Len=1410
354161	101.095556	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188865582 Ack=327 Win=62080 Len=1410
354162	101.095556	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [ACK] Seq=188866992 Ack=327 Win=62080 Len=1410
354163	101.095557	172.20.121.29	192.168.35.164	TCP	1464	8080 → 55791 [PSH, ACK] Seq=188868402 Ack=327 Win=62080 Len=1410
354220	101.276435	192.168.114.23	192.168.35.164	TCP	1514	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
354221	101.276536	172.20.121.29	192.168.35.164	TCP	1464	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
354378	101.692192	192.168.114.23	192.168.35.164	TCP	1514	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
354379	101.692341	172.20.121.29	192.168.35.164	TCP	1464	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
354632	102.520467	192.168.114.23	192.168.35.164	TCP	1514	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
354633	102.520710	172.20.121.29	192.168.35.164	TCP	1464	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
355382	104.184699	192.168.114.23	192.168.35.164	TCP	1514	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
355383	104.184894	172.20.121.29	192.168.35.164	TCP	1464	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
356560	107.544194	192.168.114.23	192.168.35.164	TCP	1514	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
356561	107.544345	172.20.121.29	192.168.35.164	TCP	1464	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
359756	114.200303	192.168.114.23	192.168.35.164	TCP	1514	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
359757	114.200443	172.20.121.29	192.168.35.164	TCP	1464	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
365922	127.511933	192.168.114.23	192.168.35.164	TCP	1514	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
365923	127.512025	172.20.121.29	192.168.35.164	TCP	1464	[TCP Retransmission] 8080 → 55791 [ACK] Seq=188679316 Ack=327 Win=62080 Len=1410
  • 192.168.35.164 is the IP of the server, in this case a pod running on a Linux node
  • 172.20.121.29 is the IP of the client
  • 192.168.114.23 is the IP of the pod in the VXLAN (from the calico IP Pool)

Additionally we observe high CPU usage on the node, but it's hard to pin down. According to some analysis the CPU usage is caused by containerd. But my analysis didn't yield any details. The issue persists even during times where the overall CPU usage is below 50% on the node.

Steps to Reproduce (for bugs)

  1. EKS Cluster running Calico and both Windows and Linux nodes
  2. HTTP server outside of the cluster / on a linux node
  3. Download a file

Context

This affects file transfering applications running on Windows nodes, that run in windows containers due to legacy reasons.

Your Environment

  • Calico version 3.27.3
  • Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.29.7-eks-2f46c53
  • Operating System and version: Windows Server 2022 Datacenter (10.0.20348.2655)
  • Link to your project (optional): -

Edit: removed ambiguity in wording
@coutinhop
Copy link
Member

@Argannor this doesn't look like a problem with Calico itself, and the symptoms look different, but from the Windows version number you're using, maybe it could be related to this Windows containers bug microsoft/Windows-Containers#516 ?

Please take a look at this issue #9019 (Windows build 20348.2582 and up seems to break networking), there is a fix on this comment: #9019 (comment)

Please let us know if that fixes it

@Argannor
Copy link
Author

Thank you for looking into this @coutinhop. We're aware of the other issue and have applied the workarounds already, which resolved that issue. I also suspect that this is not an error with calico itself.

We decided that our way forward is to abandon Windows for containerized workloads and move everything to windows. So I'll close this issue. We really don't see the production readiness of Windows Containers in general. This includes the experience in the microsoft/Windows-Containers#516 issue you already linked above. This is not to dump on Calico, in fact we're very happy with Calico :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@coutinhop @Argannor