[ard] video download results in 404 / Unable to download JSON metadata

[lnw]

• [x ] I'm reporting a broken site support
• [ x] I've verified that I'm running youtube-dl version 2021.12.17
  --> I'm running master / HEAD, but the version packaged in debian/testing has the same output
• [x ] I've checked that all provided URLs are alive and playable in a browser
• [ x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
• [x ] I've searched the bugtracker for similar issues including closed ones
  --> lots of broken URLs, but I'm fairly sure mine is correct.

Verbose log

$ youtube-dl --verbose https://www.ardmediathek.de/video/Y3JpZDovL3N3ci5kZS9hZXgvbzIwMDQ0NjI
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://www.ardmediathek.de/video/Y3JpZDovL3N3ci5kZS9hZXgvbzIwMDQ0NjI']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 40bd5c1
[debug] Python 3.11.8 (CPython x86_64 64bit) - Linux-6.6.13-amd64-x86_64-with-glibc2.37 - OpenSSL 3.1.5 30 Jan 2024 - glibc 2.37
[debug] exe versions: ffmpeg 6.1.1, ffprobe 6.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Using fake IP 53.140.55.78 (DE) as X-Forwarded-For.
[ARDBetaMediathek] Y3JpZDovL3N3ci5kZS9hZXgvbzIwMDQ0NjI: Downloading JSON metadata
ERROR: : HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
File "xxx/youtube-dl/youtube_dl/extractor/common.py", line 678, in _request_webpage
return self._downloader.urlopen(url_or_request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "xxx/youtube-dl/youtube_dl/YoutubeDL.py", line 2461, in urlopen
return self._opener.open(req, timeout=self._socket_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 525, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 634, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 563, in error
return self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/usr/lib/python3.11/urllib/request.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)

Description

I can't describe much more than what is written above. I'm not located in Germany, but I get the same error with a slightly older ytdl version from a machine that is located in D, so I don't think this is a geoblocking issue.

[dirkf]

I suspect that this is yt-dlp/yt-dlp#7666, but the withdrawn API is now getting 404 (no idea what you mean) instead of 410 (used to know this but not any longer).

[lnw]

Could be. There are some differences and I don't know the underlying structure enough to say whether it is the same underlying issue.

[dirkf]

Whichever issue at yt-dlp it was, back-porting the latest master fixes the problem:

$ python -m youtube_dl -vF 'https://www.ardmediathek.de/video/Y3JpZDovL3N3ci5kZS9hZXgvbzIwMDQ0NjI'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-vF', u'https://www.ardmediathek.de/video/Y3JpZDovL3N3ci5kZS9hZXgvbzIwMDQ0NjI']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: 625736974
[debug] Python 2.7.15 (CPython i686 32bit) - Linux-6.1.0-18-686-pae-i686-with-debian-12.5 - OpenSSL 1.1.1a  20 Nov 2018 - glibc 2.1.3
[debug] exe versions: ffmpeg 5.1.4-0, ffprobe 5.1.4-0
[debug] Proxy map: {}
[debug] Using fake IP 53.85.116.57 (DE) as X-Forwarded-For.
[ARD:mediathek] Y3JpZDovL3N3ci5kZS9hZXgvbzIwMDQ0NjI: Downloading JSON metadata
[ARD:mediathek] 13159047: Downloading m3u8 information
[info] Available formats for 13159047:
format code    extension  resolution note
http-main-0    mp4        unknown    [deu] SD , H.264
http-main-1    mp4        unknown    [deu] Full HD , H.264
hls-main-378   m3u8       480x270    [deu]  378k , avc1.4d401e, 50.0fps, mp4a.40.2
hls-main-570   m3u8       640x360    [deu]  570k , avc1.4d401f, 50.0fps, mp4a.40.2
hls-main-906   m3u8       960x540    [deu]  906k , avc1.4d401f, 50.0fps, mp4a.40.2
hls-main-1567  m3u8       1280x720   [deu] 1567k , avc1.640020, 50.0fps, mp4a.40.2
hls-main-2796  m3u8       1920x1080  [deu] 2796k , avc1.64002a, 50.0fps, mp4a.40.2
http-main-2    mp4        unknown    [deu] HD Ready , H.264
http-main-3    mp4        unknown    [deu] SD 480p , H.264 (best)
$ 

[lnw]

Indeed, the yt-dlp master (069b2aeda) works!