UDP2Raw Server Under CGNAT doen't send ACK back

[TheRainstorm]

My ISP uses Fullcone NAT, and I can retrieve my public IP after NAT using a tool called Natter. I previously set up a WireGuard tunnel using this method.

I am attempting to add a TCP layer under WireGuard, but it does not work as expected. And after debuggin, I found the udp2raw server on Host B is not sending ACK responses when receiving SYN packets from Host A.

Setup Details:

Host B run udp2raw server (wireguard listen on UDP 51821, udp2raw listen on TCP 52821)

./udp2raw_amd64 -s -l0.0.0.0:52821  -r127.0.0.1:51821   -k "xxx" --raw-mode faketcp -a

Host A run ncat, test TCP connect with B's mapped public ip after NAT

ncat -zv 36.xxx.244.22 55528

Host B udp2raw output, it shows that B get syn from A, but repeat many times

[2024-11-03 15:14:37][INFO][202.xxx.176.10:36306]received syn,sent syn ack back
[2024-11-03 15:14:38][INFO][202.xxx.176.10:36306]received syn,sent syn ack back
[2024-11-03 15:14:39][INFO][202.xxx.176.10:36306]received syn,sent syn ack back
[2024-11-03 15:14:40][INFO][202.xxx.176.10:36306]received syn,sent syn ack back
... more

Host B tcpdump, it shows that B receive A tcp sync, but no ack sent back

root@op2 ➜  tcpdump -ni any tcp port 52821
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
15:15:04.460327 pppoe-wan In  IP 202.xxx.176.10.57638 > 100.xxx.18.215.52821: Flags [S], seq 3063874040, win 64240, options [mss 1460,sackOK,TS val 1746356663 ecr 0,nop,wscale 7], length 0
15:15:05.472475 pppoe-wan In  IP 202.xxx.176.10.57638 > 100.xxx.18.215.52821: Flags [S], seq 3063874040, win 64240, options [mss 1460,sackOK,TS val 1746357xxx5 ecr 0,nop,wscale 7], length 0
15:15:06.495925 pppoe-wan In  IP 202.xxx.176.10.57638 > 100.xxx.18.215.52821: Flags [S], seq 3063874040, win 64240, options [mss 1460,sackOK,TS val 1746358699 ecr 0,nop,wscale 7], length 0
15:15:07.520559 pppoe-wan In  IP 202.xxx.176.10.57638 > 100.xxx.18.215.52821: Flags [S], seq 3063874040, win 64240, options [mss 1460,sackOK,TS val 1746359723 ecr 0,nop,wscale 7], length 0
15:15:08.543906 pppoe-wan In  IP 202.xxx.176.10.57638 > 100.xxx.18.215.52821: Flags [S], seq 3063874040, win 64240, options [mss 1460,sackOK,TS val 1746360747 ecr 0,nop,wscale 7], length 0
15:15:09.568415 pppoe-wan In  IP 202.xxx.176.10.57638 > 100.xxx.18.215.52821: Flags [S], seq 3063874040, win 64240, options [mss 1460,sackOK,TS val 1746361771 ecr 0,nop,wscale 7], length 0
15:15:11.617000 pppoe-wan In  IP 202.xxx.176.10.57638 > 100.xxx.18.215.52821: Flags [S], seq 3063874040, win 64240, options [mss 1460,sackOK,TS val 1746363820 ecr 0,nop,wscale 7], length 0

I tested this on a machine with a public IP using the same method. The expected behavior for Host B is to send an ACK back.

[wangyu-]

I cannot reproduce.
with the lastest version:

23:13:55.703545 enp1s0 In  IP 64.189.21.111.51899 > 44.55.66.77.52821: Flags [S], seq 3515756058, win 41042, options [mss 1460,sackOK,TS val 4098783012 ecr 0,nop,wscale 5], length 0
23:13:55.704099 enp1s0 Out IP 44.55.66.77.52821 > 64.189.21.111.51899: Flags [S.], seq 4214254990, ack 3515756059, win 41406, options [mss 1460,sackOK,TS val 4098782712 ecr 4098783012,nop,wscale 5], length 0
23:13:55.884157 enp1s0 In  IP 64.189.21.111.51899 > 44.55.66.77.52821: Flags [.], ack 1, win 41076, options [nop,nop,TS val 4098783194 ecr 4098782712], length 0
23:13:55.884157 enp1s0 In  IP 64.189.21.111.51899 > 44.55.66.77.52821: Flags [.], seq 1:49, ack 1, win 41286, options [nop,nop,TS val 4098783194 ecr 4098782712], length 48
23:13:55.884773 enp1s0 Out IP 44.55.66.77.52821 > 64.189.21.111.51899: Flags [.], seq 1:49, ack 49, win 41004, options [nop,nop,TS val 4098782892 ecr 4098783194], length 48
23:13:56.064830 enp1s0 In  IP 64.189.21.111.51899 > 44.55.66.77.52821: Flags [.], seq 49:97, ack 49, win 40995, options [nop,nop,TS val 4098783375 ecr 4098782892], length 48
23:13:56.065460 enp1s0 Out IP 44.55.66.77.52821 > 64.189.21.111.51899: Flags [.], seq 49:1297, ack 97, win 41054, options [nop,nop,TS val 4098783073 ecr 4098783375], length 1248
23:13:56.245985 enp1s0 In  IP 64.189.21.111.51899 > 44.55.66.77.52821: Flags [.], seq 97:1345, ack 1297, win 41312, options [nop,nop,TS val 4098783555 ecr 4098783073], length 1248
23:13:56.865754 enp1s0 Out IP 44.55.66.77.52821 > 64.189.21.111.51899: Flags [.], seq 1297:2545, ack 1345, win 41270, options [nop,nop,TS val 4098783873 ecr 4098783555], length 1248
23:13:56.886251 enp1s0 In  IP 64.189.21.111.51899 > 44.55.66.77.52821: Flags [.], seq 1345:2593, ack 1297, win 41129, options [nop,nop,TS val 4098784195 ecr 4098783073], length 1248

there can be many possible reasons:

1. maybe your way of capture doesn't work. Try to capture on interface instead of any. any sometimes doesn't work as expected
2. I see your interface is pppoe-wan. It's a special kind of interface, maybe linux rawsocket or udp2raw's code has compatiblity issuse with it.
3. maybe your server has some dual-wan setup, the reply goes to another iterface.
4. .....

[TheRainstorm]

1. Try to capture on interface instead of any
2. maybe your server has some dual-wan setup, the reply goes to another iterface.

I tried to capture on each possible interface, but the issue still persists.

2. I see your interface is pppoe-wan. It's a special kind of interface, maybe linux rawsocket or udp2raw's code has compatiblity issuse with it.

I forgot to mention that the two devices I tested are OpenWRT routers, and my broadband is fiber-optic. After the router establishes the internet connection, it creates a pppoe-wan interface.

I conducted another test, setting up a port forwarding rule on the router (B) to forward TCP traffic on port 52821 to a Debian machine (C) on the LAN. C was also running the same udp2raw command, listening on TCP port 52821. This time, I received the expected ACK. So, could the issue really be a compatibility problem with the pppoe-wan interface? I hope someone with a similar setup can reproduce this issue.

root@op2 ➜ tcpdump -ni any tcp port 52821
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
09:17:22.129257 pppoe-wan In  IP 202.xxx.176.10.51908 > 100.xxx.18.215.52821: Flags [S], seq 3626595006, win 64240, options [mss 1460,sackOK,TS val 68757137 ecr 0,nop,wscale 7], length 0
09:17:22.129308 eth1  Out IP 202.xxx.176.10.51908 > 192.168.36.187.52821: Flags [S], seq 3626595006, win 64240, options [mss 1452,sackOK,TS val 68757137 ecr 0,nop,wscale 7], length 0
09:17:22.129514 eth1  In  IP 192.168.36.187.52821 > 202.xxx.176.10.51908: Flags [S.], seq 1603582039, ack 3626595007, win 4xxx6, options [mss 1460,sackOK,TS val 4106189137 ecr 68757137,nop,wscale 5], length 0
09:17:22.129556 pppoe-wan Out IP 100.xxx.18.215.52821 > 202.xxx.176.10.51908: Flags [S.], seq 1603582039, ack 3626595007, win 4xxx6, options [mss 1452,sackOK,TS val 4106189137 ecr 68757137,nop,wscale 5], length 0
09:17:22.133328 pppoe-wan In  IP 202.xxx.176.10.51908 > 100.xxx.18.215.52821: Flags [.], ack 1, win 502, options [nop,nop,TS val 68757142 ecr 4106189137], length 0
09:17:22.133384 eth1  Out IP 202.xxx.176.10.51908 > 192.168.36.187.52821: Flags [.], ack 1, win 502, options [nop,nop,TS val 68757142 ecr 4106189137], length 0
09:17:22.133330 pppoe-wan In  IP 202.xxx.176.10.51908 > 100.xxx.18.215.52821: Flags [F.], seq 1, ack 1, win 502, options [nop,nop,TS val 68757142 ecr 4106189137], length 0
09:17:22.133418 eth1  Out IP 202.xxx.176.10.51908 > 192.168.36.187.52821: Flags [F.], seq 1, ack 1, win 502, options [nop,nop,TS val 68757142 ecr 4106189137], length 0
09:17:22.346xxx2 pppoe-wan In  IP 202.xxx.176.10.51908 > 100.xxx.18.215.52821: Flags [F.], seq 1, ack 1, win 502, options [nop,nop,TS val 68757355 ecr 4106189137], length 0
09:17:22.34xxx28 eth1  Out IP 202.xxx.176.10.51908 > 192.168.36.187.52821: Flags [F.], seq 1, ack 1, win 502, options [nop,nop,TS val 68757355 ecr 4106189137], length 0
09:17:22.554769 pppoe-wan In  IP 202.xxx.176.10.51908 > 100.xxx.18.215.52821: Flags [F.], seq 1, ack 1, win 502, options [nop,nop,TS val 68757563 ecr 4106189137], length 0
...