RST injection of 1.1.1.1:443 in China, 2023-09-05 to 2023-09-20 #285
Comments
The fact of the GFW terminating TCP connections by injecting 3 RST packets has been documented repeatedly. However, this recent injection regarding 1.1.1.1:443 is different from what I remember having been reported before. (Please correct me if I am wrong.) The main difference is that, instead of the 3 injected packets all having different sequence numbers, separated by certain intervals, in this case all 3 have the same sequence number. Also, I believe that both the client's SYN and the server's SYN+ACK triggers 3 RSTs, for 6 injected packets in total. "Ignoring the Great Firewall of China" (2006) described 3 RST packets being sent in both directions (3 to the client, 3 to the server), with sequence numbers base, base+1460, base+4380.
"Your State is Not Mine" (2017) reports something similar, which they call a "type-2 reset". The difference is that now the RST+ACK flags are set, not just RST. They also describe a "type-1 reset", which is sent 1 at a time, not 3, and does not have the ACK flag set. The spacing between sequence numbers is the same base, base+1460, base+4380 as in 2006. The "type-2 resets" are also noted to have IP TTLs and TCP window sizes that are incremental (base, base+1, base+2) within a salvo of 3.
Some other pieces of research that discuss GFW RST injection: https://censorbib.nymity.ch/#Crandall2007a, https://censorbib.nymity.ch/#Park2010a, https://censorbib.nymity.ch/#Xu2011a. The pcap I have looked at (captured at the client in China) has two noteworthy features:
As noted above, the fact that sequence numbers are the same within each salvo of 3 injected packets is different from what has been seen before. It also appears to me that both the client's SYN and the server's SYN+ACK are triggers for injection. The injected packets are sent in both directions, so each side is likely to see 3 + 3 = 6 total injected RST+ACK packets. Apart from the sequence numbers, TCP window sizes and IP TTLs are incremental within each salvo of 3, which is consistent with "Your State is Not Mine." Here is a summary of the pcap, showing 3 connection attempts:
Notice that the injected packets' seq values are equal within a salvo, and that win and ttl values are incremental (e.g. win=3909,3910,3911; ttl=145,146,147). It's also interesting that the window sizes are in the 3000–5000 range for the C→S direction, and in the 100–300 range in the S→C direction; I don't know if that pattern holds in general. At #280 (comment) it was reported that the injected RSTs' seq value could all be either 0 or a fixed value. (See the follow-up note about relative sequence numbers: what was reported as a sequence number of "1" actually means the client's initial sequence number + 1.) If my guess is right, the cases with seq=0 were triggered by the client's SYN, and the cases where seq is some other fixed value were triggered by the server's SYN+ACK. |
|
有一点需要补充，和以前不同的是，这两年我们已经能明显感受到 城市墙 的存在，不同地区，甚至同一地区的不同运营商都有不同的审查策略、执行封锁的方式，也就是说 @flowerinsnowdh 的测试目前仅代表一个地区，我们还需收集来自更多地区的反馈。 One thing to add, unlike before, in the past two years already we can clearly feel the presence of city walls, different regions and even different operators in the same region have different censorship strategies and ways of enforcing blocking, which means that @flowerinsnowdh's test is only representative of one region currently, and we still need to collect feedback from more regions. |
|
以及关于多个 RST，我有一个猜想：它们会不会是由不同层级的 GFW 分别发送的？比如市发一个、省发一个、出入境发一个。 As well as regarding multiple RSTs, I have a guess: could they be sent separately by different levels of GFW? For example, one sent by the city, one by the province, and one by the ingress/egress. |
我刚刚测试了一下用自己的网络 Wireshark 抓包，发现了变来变去无非类似下面三种情况
I just tested it with my own network Wireshark packet captures, and found out that the variation is similar to the following three cases
好像 RST 报文不能完全阻断 TCP 连接，有时服务端也能成功发来 SYN ACK，RST 有时也指向了不同的目标，但是没有访问成功的案例，最终都会显示为连接重置 It seems that the RST message does not completely block the TCP connection, sometimes the server can also successfully send SYN ACK, RST sometimes also points to a different target, but there is no access to a successful case, will eventually show up as a connection reset |
|
I received two reports that match the pattern I described above: 3 RST+ACKs in response to the client's SYN, 3 RST+ACKs in response to the server's SYN+ACK, sequence numbers identical within a group of 3, incremental TTL and window size, no packet dropping. But I also saw one other pcap that was a little different. In response to the client's SYN, there was only 1 RST+ACK, not 3, and there was some kind of packet dropping happening, because no SYN+ACK was received from the server. However, in a small number of cases, the client's SYN got through with no injection, and in those cases, the server's SYN+ACK elicited the same pattern of multiple RST+ACKs with incremental TTL and window size as described earlier. (I saw a case with 2 RST+ACKs and one with 3 RST+ACKs in response to the server's SYN+ACK.) In that pcap, it looks as though there are two different filters operating separately in each direction:
This other pcap, with different filtering in the C→S and S→C directions, matches the first two examples in what @flowerinsnowdh posted immediately above.
The third of the three examples is a little different.
So it appears at least 3 behaviors may be possible:
|
In the case of the salvos of 3 RST+ACKs, I think it's unlikely that they sent by separate systems, because the arrival timestamps are so close (equal down to the millisecond in #285 (comment) and #285 (comment)), and because the incremental IP TTL and TCP window size would be difficult to synchronize across different injection systems. But in the case just above where it seems like there's different injection behavior in the C→S and S→C directions (1 RST+ACKs versus 3, packet dropping versus no), there I can easily believe that they are separate injection systems, based on what I have seen so far. |
|
Timestamps 非常相近的情况，还有可能是不同层级的 GFW 把封锁策略部署到了边缘（为了减轻出入境的压力等），比如说每一层都想给 https://1.1.1.1 发个 RST，而边缘没有先把这些策略合并，只是依次执行，如果有网站是两个 RST 则这一猜想更有可能为真 In cases where the timestamps are very similar, it is also possible that different tiers of GFW have deployed blocking policies to the edge (to take pressure off the entry and exit points, etc.), e.g. each tier is trying to send an RST to https://1.1.1.1 and the edge is just executing them sequentially without merging them first, which is a more likely scenario if there is a site with two RSTs |
Why would it use such a method? Packet dropping would imply some buffering of packets before decision can be made about them, which is more costly than injecting resets from tapping firewalls. Also, in my testing the single RST/ACKs injected in response to SYNs have their mean RTTs approximately the same or slightly more than the RTT of the last hop in China, meaning these single RST/ACKs are still sent from near the last hop instead of some kind of edge nodes where inband firewalls can do more heavy duty blcoking. Another interesting tidbit: I use this one-liner In this case the connection is totally unmolested by resets. But if I use |
curious if you are you reusing the same client port number here? |
I agree, it doesn't make sense for there to be both RST injection and packet dropping. It was suggested to me that the SYN+ACK might be getting dropped by a local NAT, rather than the GFW. The injected RST removes the NAT mapping, so the SYN+ACK gets dropped. Something like this:
|
No, at least not shown in Wireshark. Edit: My testing point is behind a CGNAT so it is possible the port translated by the CGNAT and visible to the GFW is reused. If that is the case, then it's possible the GFW also maintains a regular connection lifetime but ignores the rest of the connection after resets on both directions while the CGNAT reuses the port for a new connection. |
You might be able to test this out by spamming SYNs from the client through the NAT. As long as a SYN makes it after the RST+ACK and before the SYN+ACK and matches the TCP connection (i.e., ports, ips, SEQ/ACK_SEQ), the SYN/ACK may reach the client. I'd expect so any way for something Linux or freebsd-like. |
something along those lines was also my thinking, and it would make sense together with the observation made in #275 -- the censorship system in the paper additionally uses sequence numbers to identify TCP sessions, and I suspect it is because they cannot maintain a strongly-consistent session table/flow tracking table in a central place due to the scale they're operating on. if there was a strongly-consistent table then I would have expected the SYN after the RST to actually be identified as a new connection. assuming you have no way to determine which client port is visible outside of CGNAT (maybe cloudflare can reflect it for you or you try another server that does that), I wonder if you are hitting a race condition in flow tracking, and putting one thing I cannot wrap my head around is if all of this is consistent or inconsistent with @wkrp's findings |
The CGNAT in my testing cycles through 4096 externally visible ports per IP. But this seems to have no correlation with the frequency of getting a The following is just for fun. I use this script to collect some survival rates if there is some wait after connecting: import socket
import time
import sys
from collections import defaultdict
if len(sys.argv) != 3:
print(f'Usage: f{sys.argv[0]} clienthello.bin wait_after_connect')
sys.exit(1)
clienthello = open(sys.argv[1], mode='rb').read()
wait_after_connect = float(sys.argv[2])
count = defaultdict(int)
for i in range(10000):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
try:
s.connect(('1.1.1.1', 443))
except ConnectionRefusedError as e:
continue
except ConnectionResetError as e:
continue
time.sleep(0.1)
try:
s.recv(1, socket.MSG_PEEK | socket.MSG_DONTWAIT)
except ConnectionResetError as e:
continue
except BlockingIOError as e:
pass
time.sleep(wait_after_connect)
try:
s.sendall(clienthello)
except ConnectionResetError as e:
print(i, 'reset during send')
count['reset during send'] += 1
continue
data = b''
s.settimeout(0.5)
reset = False
while True:
try:
data += s.recv(10240)
except TimeoutError as e:
break
except ConnectionResetError as e:
print(i, 'reset after send')
count['reset after send'] += 1
reset = True
break
if reset:
continue
data_descr = f'{len(data)} bytes'
print(i, data_descr)
count[data_descr] += 1
print(dict(count))and have these results: Increasing the delay after connecting doesn't improve the survival rate after higher than 1 second. It seems there is some temporal locality in congestion(?) instead of just connection expiration in flow tracking.
For this: |
|
此前我们观察到 GFW 发的 RST 包与正常包的 TTL 有明显差异，或许我们可以简单地丢弃 TTL 异常的 RST 包， 不过该方法对双向 RST 不太好使（如果目标网站不受控），以及境内段层层 NAT 可能会响应 RST，但是值得测试一下 Previously we observed that the RST packets sent by GFW had a significantly different TTL than normal packets, so perhaps we could simply discard the RST packets with the abnormal TTL However, this method doesn't work well for bi-directional RST (if the target site is uncontrolled), and the layers of NAT in the internal segments may respond to RST, but it's worth a test |
|
as you said, it also is not clear to me how any NAT alongside the terminated connection can be convinced to ignore the RST it is likely that between censor and server, there is no NAT. I wonder if it is somehow possible to 1) drop RST on the server as you describe 2) re-connect from the server to the client using NAT traversal https://tailscale.com/blog/how-nat-traversal-works/ is very interesting however, generally i feel that all these tricks are full of detectable features, and at the same time fragile against network topology, so maybe the entire line of thinking is a dead end in general (i.e. we should accept the RST when it comes and instead try not to trigger it, as we already do) |
是这样没错，这些小 trick 就只是像 @gfw-report 的那些方法一样， 就像很多伊朗朋友希望 Xray 加的 各种分片，我觉得它们特征明显，但有时就是能用，以及最近 @5e2t 又继续测试的 河南 RST That's right, these little tricks are just a way to Like the various fragments that many of my Iranian friends want Xray to add, which I think are obvious, but sometimes just work, and the Henan RST that @5e2t has continued to test recently. |
|
其实这里提到的 3 个 RST 看起来基本上是 GFW 特意设置的冗余策略，但我一直在想有没有一种可能，一个数据包会依次受到多级 GFW（或多个未经合并的规则）的审查、触发多次操作。 比如，@5e2t 发现他那里（河南）访问某些域名会收到 4 个 RST，其中一个 RST 与众不同：XTLS/Xray-core#2426 (comment) In fact, the 3 RSTs mentioned here seem to be basically a deliberate redundancy policy set up by GFW, but I've been wondering if it's possible for a single packet to be scrutinized by multiple levels of GFW (or multiple unconsolidated rules) in sequence, triggering multiple actions. For example, @5e2t found that he was getting 4 RSTs for accessing certain domains there (Henan), one of which was different: XTLS/Xray-core#2426 (comment) |
Yes, the fact that there are 3 RSTs is not surprising (to me): it's been that way for as long as I can remember, at least as far back as 2006. The thing that seemed unusual to me, this time, is that the sequence numbers in the RST were equal, instead of being spaced out as has been documented before.
For DNS injection, at least, it is certainly the case that there are multiple middleboxes that independently inject false DNS responses. See the "Triplet Censors". It is possible to get 2 or 3 injected DNS responses for a single DNS query.
I did a check just now to see what kind of RSTs I would get for simple HTTP Host censorship. It's easy to test because it's bidirectional. I used this command to send a request for a blocked domain name through the firewall from the outside (www.cqa.cn is the web site of an airport in China): Interestingly, from this command I got seven RSTs, with apparently three different network fingerprints (note flags, win, ipid, ttl). It looks like a group of 1, a group of 3, and another group of 3. All of the sequence numbers were equal. TCP window sizes and IP TTLs were not incremental.
Maybe the "same sequence numbers" feature is not new with 1.1.1.1; maybe the sequence numbers stopped being spaced out some time in the past and it just wasn't documented. |
|
I got a report that RST injection of 1.1.1.1:443 has stopped as of 2023-09-20. What do you see? |
|
Stopped here too. |
现在 https://1.1.1.1/ 可以访问了，https://8.8.8.8/ 仍不可访问 Now https://1.1.1.1/ is accessible, https://8.8.8.8/ is still inaccessible |
wkrp
changed the title
当时我试了，可以访问了，但最近有报告称它又被针对了 #292 (comment) ，你们测测看。 I tried it at the time and was able to access it, but recently there have been reports of it being targeted again #292 (comment), test it out guys. |
|
@RPRX
|
|
@RPRX The traffic between me and 1.1.1.1 is through Shanghai exit GFW, I get RST in 30ms, the delay between me and Shanghai exit GFW is also 30ms. |
This thread is to consolidate reports from other threads, particularly #280 (comment) and #284, as well as information received privately.
In summary: since 2023-09-05 (about 06:00 UTC), there is new RST injection of https://1.1.1.1/ (which, notably, is a DNS over HTTPS resolver. 1.1.1.1 may have been blocked in other ways in some regions before; but there was definitely a change starting 2023-09-05. The RST injection resembles some of the "classic" RST injection that has been documented in China in the past, but there are some differences. It also differs from the well-known SNI filtering, in that the injection is triggered by the client's TCP SYN packet, not the TLS Client Hello.
Blocking of 8.8.8.8:443 (Google's DNS over HTTP resolver) is unrelated. It is blocked by packet dropping / null routing, not RST injection (ref).
The text was updated successfully, but these errors were encountered: