Throttling→blocking of YouTube in Russia, 2024-07-12

[wkrp]

There are two long threads at NTC about throttling of YouTube video playback in Russia since 2024-07-12.

Technical thread: Замедление YouTube в России YouTube slowdown in Russia
Discussion thread: Обсуждение: Замедление YouTube в России Discussion: YouTube slowdown in Russia

From what I can gather, the throttling is triggered by an SNI match of *.googlevideo.com in TLS or QUIC. (Which indicates it is targeted at video download/playback, not the web UI of youtube.com.) You can use a different SNI with a Google server and avoid throttling, or use a googlevideo.com with a non-Google server and still get throttled. The prevalence of throttling is not uniform (see clarification) throughout the country, and has apparently been expanding progressively. The throttling is being done on TSPU devices by packet dropping, perhaps of client→server ACK packets. The effect on performance is variable and depends on a number of technical and configuration factors: web browser, curl, or yt-dlp; HTTP/1.1, HTTP/2, or HTTP/3 (QUIC); and specifics of the browser or TLS stack. At first, starting 2024-07-12, only TCP-based transfers were affected (HTTP/1.1 and HTTP/2) and QUIC was not affected. On 2024-07-18, QUIC started to be affected.

The slowdown of YouTube was acknowledged from the beginning, but at first the government and ISPs claimed that the cause was high demand and inadequate maintenance of Google infrastructure, such as Google Global Cache. See, for example, this 2024-07-12 Meduza article (archive) and this Rostelecom statement (archive):

«Ростелеком» информирует о наличии технических проблем в работе оборудования, принадлежащего компании Google и используемого на сетевой инфраструктуре оператора и пиринговых стыках. Данное оборудование применяется для кэширования и ускорения загрузки контента сервисов Googlе, в основном видеохостинга YouTube (система Google Global Cache).

Из-за проблем с эксплуатацией указанного оборудования и невозможности его расширения в условиях роста обрабатываемого трафика, наблюдается серьезная перегрузка имеющихся мощностей, в том числе на пиринговых стыках. Это может повлиять на скорость загрузки и качество воспроизведения роликов в YouTube у абонентов всех российских операторов.

Rostelecom informs about the availability of technical problems in the operation of equipment belonging to Google and the operator used on the network infrastructure and peer joints. This equipment is used to caching and accelerating the loading of Google services, mainly YouTube video hosting (Google Global Cache system).

Due to the problems with the operation of the indicated equipment and the impossibility of expanding it in the conditions of growth of processed traffic, there is a serious overload of the existing capacities, including peer joints. This can affect the speed of loading and the quality of playback of rollers in YouTube from subscribers of all Russian operators.

But just today, authorities acknowledged that the throttling was intentional, with a target transfer rate of 128 kbit/s.

Российские власти уведомили провайдеров о замедлении ютьюба до 128 килобит в секунду (archive) Russian authorities notified providers about the slowdown of YouTube to 128 kilobits per second
Russian authorities inform local Internet service providers that most YouTube access is now capped at a piddling 128 kbps (archive)

Российские власти уведомили всех крупнейших операторов связи в стране о замедлении скорости воспроизведения видео на ютьюбе до 128 килобит в секунду. Об этом «Медузе» рассказал источник на телекоммуникационном рынке. В июле этот же собеседник «Медузы» одним из первых сообщил, что замедление ютьюба в России связано с техническими действиями Роскомнадзора, а не с износом серверов Google, как сообщал «Ростелеком» (позднее эту версию подтвердили и независимые эксперты).

Скорости в 128 килобит в секунду должно хватить для нормального прослушивания аудио или просмотра видео в разрешении не выше 240p. При выборе видео с более высоким разрешением пользователи скорее всего столкнутся с ощутимыми задержками при загрузке.

О фактически введенном в столице лимите в 128 килобит в секунду для ютьюба 1 августа написал и ValdikSS — анонимный разработчик систем обхода блокировок: «Ровно в полночь по Москве включили замедление по TCP до ≈128 кбит/с, как это было с Twitter, почти везде без burst, а QUIC до порядка ≈512 кбит/с».

По словам источника «Медузы», власти пока решили «не выключать [ютьюб] полностью, но сильно замедлить». «Всем операторам связи задали рекомендуемый уровень [скорости воспроизведения роликов на платформе] — 128 килобит в секунду на домашнем интернете», — уточнил собеседник.

Russian authorities have notified all major telecom operators in the country about slowing down the speed of video playback on YouTube to 128 kilobits per second. A source in the telecommunications market told Meduza about this. In July, the same Meduza interlocutor was one of the first to report that the slowdown of YouTube in Russia was due to technical actions of Roskomnadzor, and not due to the wear and tear of Google's servers, as reported by Rostelecom (this version was later confirmed by independent experts).

The speed of 128 kilobits per second should be enough for normal listening to audio or watching video in a resolution not higher than 240p. When choosing a video with a higher resolution, users are likely to encounter noticeable delays in downloading.

ValdikSS, an anonymous developer of blocking circumvention systems, wrote about the limit of 128 kilobits per second actually introduced in the capital city for YouTube on August 1: "At exactly midnight in Moscow, we switched on TCP slowdown to ≈128 kbps, as it was with Twitter, almost everywhere without burst, and QUIC to about ≈512 kbps.

According to Meduza's source, the authorities have so far decided "not to turn [YouTube] off completely, but to slow it down a lot." "All telecom operators have been set the recommended level [of speed for playing videos on the platform] - 128 kilobits per second on the home Internet," the interlocutor said.

Past documentation on throttling of twitter.com and other sites in Russia in 2021 and 2022:

• Throttling of Twitter domains in Russia #65
• Partial blocking or throttling of social media in Russia #106
• Throttling Twitter: An Emerging Censorship Technique in Russia

[fortuna]

Thanks for the excellent summary.

The question is: how can one bypass the interference?
It looks like domain fronting works?
What else?

[wkrp]

It seems that GoodbyeDPI defaults are sufficient. But you likely also have to disable Kyber key exchange in the browser, because Kyber may push the SNI into a second TCP segment, where TSPU will find it but GoodbyeDPI currently does not.

https://ntc.party/t/8074/41 2024-07-15

Запустите goodbyedpi без параметров. frag-by-sni в России не нужен.
Не забудьте отключить Kyber в браузере.

Run goodbyedpi without parameters. frag-by-sni is not needed in Russia.
Don't forget to disable Kyber in your browser.

EDIT: Although I'm not sure, this post from 2024-08-02 says that TSPU had started blocking connections to googlevideo.com IP addresses from which it is not able to extract an SNI.

Похоже что теперь соединения c *.googlevideo.com в которых ТСПУ не находит sni дропаются. Проверял сравнивая результат трех команд:

openssl s_client -connect one_of_subdomains.googlevideo.com:443
openssl s_client -connect one_of_subdomains.googlevideo.com:443 -noservername
openssl s_client -connect one_of_subdomains.googlevideo.com:443 -noservername -proxy my_proxy

Первая и третья успешно устанавливают соединение. Вторая не получает ответа на client hello и шлет ретраи.

It seems that now the connections with *.googlevideo.com in which the TSPU does not find sni are dropped. I checked by comparing the result of three commands:

openssl s_client -connect one_of_subdomains.googlevideo.com:443
openssl s_client -connect one_of_subdomains.googlevideo.com:443 -noservername
openssl s_client -connect one_of_subdomains.googlevideo.com:443 -noservername -proxy my_proxy

The first and third successfully establish the connection. The second does not receive a response to client hello and sends retries.

[ValdikSS]

@wkrp, great summary, you're doing the work I have no time for. Let me add and correct you just a bit:

1. Throttling is uniform in general (*.googlevideo.com towards irrelevant hosts is throttled ≈everywhere), however some hosts, such as GGC hosted on ISP premises (and on ISP IP ranges), may not get throttled. In case of my ISP, ISP-local GGC IPv4 address is not throttled while IPv6 address is (both were not throttled in the first days of August). This is the main reason people do not experience throttling — many ISP-local GGCs which the filter ignores for some reason, probably due to oversight.
2. At some point some GGCs were throttled by IP range, irrelevant of SNI. This was the case of Tattelecom ranges. Just tested these right now — not throttled by IP anymore, but from my home ISP the server on rr1---… responds only 1 time out of 5 connections (connection hangs after ClientHello, irrelevant of SNI). No such issue on the other internet link to the same server.
3. There may be dynamic blocking upon some trigger (censorship circumvention), see this, where the hosts became suddenly unreachable (TLS hangs as in the previous point).
4. At some point, Firefox's QUIC was not throttled, probably because its ClientHello is slightly larger than the other browsers. Now it is throttled.
5. User-Agent (the software, not the header) is important: YouTube by itself is sophisticated, it connects to different GGCs, could try the others (city-global/region-global I believe) if ISP-local is not reachable or if connection instabilities are detected. It also could jump from one server to another, especially on seek, which gives speed boost due to traffic burst.
   I haven't researched mobile applications much, but I guess they have even more quirks for smooth video playback which partially mitigate the issue.
6. Chrome uses QUIC "connection migration" which is basically a connection teardown and resumption using another source port and without handshake, so the DPI can't detect SNI in it. Such "resumed" connections were not throttled when I checked it last time (beginning of last week). They have enabled blocking of fully-encrypted protocols, maybe to block such connections, or maybe on unrelated note.
7. QUIC is throttled less, probably thanks to its more robust congestion control algorithm for a crude traffic policing.

[gamer191]

SUBDOMAIN.googlevideo.com seems to be mirrored at SUBDOMAIN.c.drive.google.com
Do you know whether the latter is throttled?

[ValdikSS]

@gamer191

Nah, unfortunately GGCs don't have certificate for this domain.

Websites prove their identity via certificates. Firefox does not trust this site because it uses a certificate that is not valid for rr2---sn-capm-vnae.c.drive.google.com. The certificate is only valid for the following names: *.googlevideo.com, *.a1.googlevideo.com, *.a1.googlevads-cn.com, *.bdn.dev, *.origin-test.bdn.dev, *.c.doc-0-0-sj.sj.googleusercontent.com, *.c.2mdn.net, *.c.2mdn-cn.net, *.dai.googlevideo.com, *.dai.googlevads-cn.com, *.googlezip.net, *.gvt1.com, *.gvt1-cn.com, *.offline-maps.gvt1.com, *.snap.gvt1.com, *.snap.gvt1-cn.com, *.gcpcdn.gvt1.com, xn--ngstr-lra8j.com, *.xn--ngstr-lra8j.com, i.ytimg.com, play-lh.googleusercontent.com

[gamer191]

@ValdikSS I'm getting that as well now, which is quite strange. It seems like some subdomains are using a different certificate to others
However, your message gave me an idea, which was to change the url to SUBDOMAIN.c.doc-0-0-sj.sj.googleusercontent.com or SUBDOMAIN.xn--ngstr-lra8j.com. Note that the second domain is written in punycode, so it's equivalent to SUBDOMAIN.ångströ.com

I assume more of the domains listed in your message work, but I don't have time to test them right now
Can you please check if either of the domains I've listed are unthrottled in Russia?

Side note: the mn= parameter lists a fallback subdomain after the %2C, so it doesn't matter if one subdomain fails (source: yt-dlp/yt-dlp#1986)

EDIT: SUBDOMAIN.c.2mdn.net also works, although that domain may trigger some adblockers

[wkrp]

Today there is news and user reports that YouTube is no longer just throttled but blocked, with at least the following domain names now being blocked by SNI:

youtube.com
ytimg.com
youtubei.googleapis.com (but not other *.googleapis.com)

The posts at NTC that report the blocking start at around 2024-08-08 07:00.

• 2024-08-08 06:53

  На TTK ютуб полностью умер. Даже страницы по 2-5 минут загружаются. И 144p виснет навсегда.

  On TTK, YouTube is completely dead. Even pages take 2-5 minutes to load. And 144p hangs forever.

• 2024-08-08 07:23

  youtube.com не открывается вообще
  ytimg.com замедляют

  youtube.com does not open at all
  ytimg.com slow down

Meduza has stories in Russian and English, with user testimonials and workarounds.

YouTube перестал открываться в России (archive)
YouTube ceased to open in Russia

Видеохостинг YouTube перестал открываться на компьютерах и мобильных устройствах у пользователей в России утром 8 августа, следует из данных сервиса Brand Analytics.

YouTube video hosting has ceased to open on computers and mobile devices for users in Russia on the morning of August 8, according to Brand Analytics service data.

«Он мертв, не грузит вообще» Читатели «Медузы» — о замедлении (а может, и блокировке) ютьюба. И о том, какими методами они с этим справляются (есть полезные советы) (archive)

Но все может измениться в худшую сторону уже очень скоро: утром 8 августа пользователи со всей страны начали массово жаловаться на то, что ютьюб перестал открываться вовсе — причем не только в десктопной версии, но и на мобильных устройствах. Началась ли в стране блокировка сервиса и как долго она продлится, неизвестно. А пока мы решили поделиться ответами читателей «Медузы», которые рассказали нам, как борются с проблемами в работе ютьюба.

YouTube has suddenly stopped working in Russia. Meduza’s readers describe how they’re handling the loss of the world’s most popular video streaming service. (archive)

That could change today, however, following reports from across Russia on the morning of August 8 that Google’s streaming service has stopped openly entirely — not only on desktop computers but also on mobile devices. At the time of this writing, it’s still unclear if Russia is now blocking YouTube outright and how long this might last. Meduza has collected accounts of the YouTube outage from readers in Russia. Below, you can read about how ordinary Russians are dealing with the sudden and total loss of the world’s most popular video streaming service.

[fortuna]

I ran some measurements. Blocking seems to happen on residential ISPs only. Here are the results:

Data: results_direct.csv

This was done measuring successful TLS handshakes over TCP only. QUIC was not tested. IPs were pre-resolved, DNS was not tested.

[fortuna]

Upon further investigation, I believe the inconsistent results for some domains on the residential networks are the effect of throttling, not blocking.

Below we look at 4 measurements of www.youtube.com on Bee Line Home (AS8402). The two on the left are successful, while the two on the right are failures.

The JSON files show network actions, like read and write, during the TLS handshake. Field t is the time in seconds since the connection started.

In this case, they all successfully completed the TLS handshake. The timeout happened in the data transfer part ot the connection.

However, notice how the the first read took way longer on examples #3 and #4. On #4, the first read took over 1.4s, and the third read took 2s!

This is an indication that the handshake is being throttled. Likely packets are being dropped or held.

The total times to complete the TLS handshakes were 0.43s, .65s, 1.32s and 5s.

Besides the long time to complete reads, the failure cases are also reading less data in each read than in the success cases. This aligns with TCP typical behavior in reaction to packet drops.

Measurements:

• https://explorer.ooni.org/m/20240816131845.085577_RU_webconnectivity_296bf3ec82c2bfb0
• https://explorer.ooni.org/m/20240816134229.165983_RU_webconnectivity_6266ed1e0d0138aa
• https://explorer.ooni.org/m/20240816131928.699638_RU_webconnectivity_a8f5a9fdd97aec6d
• https://explorer.ooni.org/m/20240816133140.968412_RU_webconnectivity_6f2feae5919d46ec

You can see www.youtube.com for the 5 ISPs I analyzed on the OONI Explorer:

[shikantazacomputers]

Some context from a Russian YouTuber: https://www.youtube.com/watch?v=DK16f-WEpRQ

[RPRX]

Chrome uses QUIC "connection migration" which is basically a connection teardown and resumption using another source port and without handshake, so the DPI can't detect SNI in it.

说起来，中国并没有封死 Cloudflare，而是仅阻断部分 SNI，以及有时会阻断 WSS 等有代理特征的连接，对它的 QUIC 似乎没有明显封锁，那么我有个大胆的想法是手动 ECH：针对被代理的 QUIC 连接只代理握手，剩下的可以直连 Cloudflare 同一 IP

Speaking of which, China doesn't block Cloudflare, but only blocks some SNIs, and sometimes WSS and other proxy-characterized connections, and there doesn't seem to be any significant blocking of its QUIC, so I've got a bold idea for a manual ECH: Proxy only the handshake for proxied QUIC connections, and then the rest can be directly connected to the same IP as Cloudflare.

[aliakseis]

This is from Copilot:

To connect to an HTTPS resource using a SAN (Subject Alternative Name) certificate with curl, you can specify the IP address for the connection and set the Host header to match the domain name in the SAN certificate. Here's how you can do it:

Step-by-Step Guide

1. Use the --resolve option:
   • This option allows you to specify the IP address for a given hostname, ensuring that curl connects to the correct server while using the hostname for the SSL/TLS handshake.
2. Set the Host header:
   • The Host header ensures that the server recognizes the request as intended for the specified domain.

Example Command
curl --resolve example.com:443:192.168.1.1 https://example.com/resource -o output.txt

Using this approach in my experimental YouTube player with modified PyTube, everything works as expected, but there are no SNI values.

[ValdikSS]

Roscomnadzor has blocked other SNI to GGC, apparently to prevent GoodbyeDPI/zapret from working with default configuration. www.google.com works fine for now, as well as pushing random traffic pattern (non-TLS) before connection.

Some points got the entire TLS traffic blocked, regardless of SNI.