Skip to content

Releases: devine-dl/devine

v3.3.3

07 May 06:11
Compare
Choose a tag to compare

Bug Fixes

  • dl: Automatically convert TTML Subs to WebVTT for MKV support
  • Subtitle: Correct timestamps when merging fragmented WebVTT

Changes

  • env: List all directories as table in info
  • env: List possible config path locations when not found
  • binaries: Move all binary definitions to core/binaries file
  • curl-impersonate: Remove manual fix for curl proxy SSL
  • curl-impersonate: Update the default browser to chrome124
  • Config: Move possible config paths out of func to constant
  • utilities: Remove get_binary_path, use binaries.find instead
  • dl: Improve readability of download worker errors
  • env: Shorten paths on Windows with env vars

v3.3.2

16 Apr 05:07
Compare
Choose a tag to compare

Bug Fixes

  • Video: Ensure track is supported in change_color_range()
  • Video: Optionalise constructor args, add doc-string & checks
  • Audio: Optionalise constructor args, add doc-string & checks
  • Subtitle: Optionalise constructor args, add doc-string & checks
  • HLS: Ensure playlist.stream_info.codecs exists before use
  • HLS: Ensure playlist.stream_info.resolution exists before use
  • env: List used config path, otherwise the default path
  • cfg: Use loaded config path instead of hardcoded default
  • Basic: Return None not Exception if no proxy configured

Changes

  • Video: Do not print "?"/"Unknown" values in str()
  • Audio: Do not print "?"/"Unknown" values in str()
  • Subtitle: Do not print "?"/"Unknown" values in str()
  • Audio: List lang after codec for consistency with other Tracks
  • Video: Return None if no m3u RANGE, not SDR
  • env: Use -- to indicate no config found/loaded

New Contributors

v3.3.1

05 Apr 11:32
Compare
Choose a tag to compare

Features

  • dl: Add new --workers to set download threads/workers

Bug Fixes

  • Chapter: Cast values to int prior to formatting
  • requests: Fix multithreaded downloads
  • Events: Dereference subscription store from ephemeral store

Changes

  • dl: Change --workers to --downloads

New Contributors

v3.3.0

02 Apr 21:00
Compare
Choose a tag to compare

Features

  • Add support for MKV Attachments via Attachment class
  • dl: Automatically attach fonts used within SSAv4 subs
  • dl: Try find SSAv4 fonts in System OS fonts folder
  • Basic: Allow single string URIs for countries
  • Basic: Allow proxy selection by index (one-indexed)
  • Events: Add new global Event Observer API

Bug Fixes

  • curl-impersonate: Set Cert-Authority Bundle for HTTPS Proxies
  • Basic: Make query case-insensitive
  • WVD: Ensure WVDs dir exists before moving WVD file
  • WVD: Fix empty path to WVDs folder check
  • WVD: Move log out of loop to save performance
  • WVD: Move log with path before Device load
  • WVD: Add exists/empty checks to WVD folder dumps
  • Basic: Fix variable typo regression

Changes

  • Basic: Improve proxy format checks
  • WVD: Print error if path to parse doesn't exist
  • WVD: Seperate logs in loop for visual clarity
  • Track: Move from OnXyz callables to Event observer

v3.2.0

25 Mar 03:55
Compare
Choose a tag to compare

Features

  • ClearKey: Pass session not proxy str in from_m3u_key method
  • Track: Allow Track to choose downloader to use
  • search: New Search command, Service method, SearchResult Class

Bug Fixes

  • dl: Include chapters when muxing
  • aria2c: Support aria2(c) 1.37.0 by handling upstream regression
  • MultipleChoice: Simplify super() call and value types
  • dl: Add single mux job if there's no video tracks
  • Track: Compute Track ID from the this variable, not self
  • DASH/HLS: Don't merge folders, skip final merge if only 1 segment
  • dl: Use click.command() instead of click.group()
  • HLS: Remove save dir even if final merge wasn't needed
  • Track: Fix order of operation mistake in get_track_name
  • requests: Set HTTP pool connections/maxsize to max workers
  • Video: Delete original file after using change_color_range()
  • Video: Delete original file after using remove_eia_cc()
  • requests: Manually compute default max_workers or pool size is None
  • requests: Block until connection freed if too many connections
  • HLS: Delete subtitle segments as they are merged
  • HLS: Delete video/audio segments after FFmpeg merge

Changes

  • ClearKey: Only use User-Agent if none set in from_m3u_key
  • Track: Remove TERRITORY_MAP constant, trim SAR China manually
  • Track: Default the track name to it's lang's script/territory
  • Service: Go back to the default pool_maxsize in Session

v3.1.0

05 Mar 17:13
Compare
Choose a tag to compare

Features

  • cli: Implement MultipleChoice click param based on Choice param
  • dl: Skip video lang filter if --v-lang unused & only 1 video lang
  • dl: Change --vcodec default to None, use any codec
  • dl: Support multiple -r/--range and mux ranges separately
  • Subtitle: Convert from fTTML->TTML & fVTT->WebVTT post-download
  • Track: Make ID optional, Automatically compute one if not provided
  • Track: Add a name property to use for the Track Name

Bug Fixes

  • dl: Have --sub-format default to None to keep original sub format
  • HLS: Use filtered out segment key info
  • Track: Don't modify lang when getting name
  • Track: Don't use fallback values "Zzzz"/"ZZ" for track name
  • version: The __version__ variable forgot to be updated

Changes

  • Move dl command's download_track() to Track.download()
  • dl: Remove unused get_profiles() method
  • DASH: Move data values from track url to track data property
  • DASH: Change how Video FPS is gotten to remove FutureWarning log
  • Track: Add type checks, improve typing
  • Track: Remove swap() method and it's uses
  • Track: Remove unused DRM enum
  • Track: Rename Descriptor's M3U & MPD to HLS & DASH
  • Track: Remove unnecessary bool casting
  • Track: Move the path class instance variable with the rest
  • Track: Return new path on move(), raise exceptions on errors
  • Track: Move delete and move methods near start of Class
  • Track: Rename extra to data, enforce type as dict

Builds

  • Explicitly use marisa-trie==1.1.0 for Python 3.12 wheels

v3.0.0

01 Mar 00:04
Compare
Choose a tag to compare

Added

  • Support for Python 3.12.
  • Audio track's Codec Enum now has FLAC defined.
  • The Downloader to use can now be set in the config under the downloader key.
  • New Multi-Threaded Downloader, requests, that makes HTTP(S) calls using Python-requests.
  • New Multi-Threaded Downloader, curl_impersonate, that makes HTTP(S) calls using Curl-Impersonate via Curl_CFFI.
  • HLS manifests specifying a Byte range value without starting offsets are now supported.
  • HLS segments that use EXT-X-DISCONTINUITY are now supported.
  • DASH manifests with SegmentBase or only BaseURL are now supported.
  • Subtitle tracks from DASH manifests now automatically marked as SDH if urn:tva:metadata:cs:AudioPurposeCS:2007 = 2.
  • The --audio-only/--subs-only/--chapters-only flags can now be used simultaneously. For example, --subs-only
    with --chapters-only will get just Subtitles and Chapters.
  • Added --video-only flag, which can also still be simultaneously used with the only "only" flags. Using all four
    of these flags will have the same effect as not using any of them.
  • Added --no-proxy flag, disabling all uses of proxies, even if --proxy is set.
  • Added --sub-format option, which sets the wanted output subtitle format, defaulting to SubRip (SRT).
  • Added Subtitle.reverse_rtl() method to use SubtitleEdit's /ReverseRtlStartEnd functionality.
  • Added Subtitle.convert() method to convert the loaded Subtitle to another format. Note that you cannot convert to
    fTTML or fVTT, but you can convert from them. SubtitleEdit will be used in precedence over pycaption if available.
    Converting to SubStationAlphav4 requires SubtitleEdit, but you may want to manually alter the Canvas resolution after
    the download.
  • Added support for SubRip (SRT) format subtitles in Subtitle.parse() via pycaption.
  • Added API Vault Client aiming for a RESTful like API.
  • Added Chapters Class to hold the new reworked Chapter objects, automatically handling stuff like order of the
    Chapters, Chapter numbers, loading from a chapter file or string, and saving to a chapter file or string.
  • Added new chapter_fallback_name config option allowing you to set a Chapter Name Template used when muxing Chapters
    into an MKV Container with MKVMerge. Do note, it defaults to no Chapter Fallback Name at all, but MKVMerge will force
    Chapter {i:02} at least for me on Windows with the program language set to English. You may want to instead use
    Chapter {j:02} which will do Chapter 01, Intro, Chapter 02 instead of Chapter 01, Intro, Chapter 03 (an Intro
    is not a Chapter of story, but it is the 2nd Chapter marker, so It's up to you how you want to interpret it).
  • Added new Track.OnSegmentDownloaded Event, called any time one of the Track's segments were downloaded.
  • Added new Subtitle.OnConverted Event, called any time that Subtitle is converted.
  • Implemented __add__ method to Tracks class, allowing you to add to the first Tracks object. For example, making
    it handy to merge HLS video tracks with DASH tracks, tracks = dash_tracks + hls_tracks.videos, or for iterating:
    for track in dash.videos + hls.videos: ....
  • Added new utility get_free_port() to get a free local port to use, though it may be taken by the time it's used.

Changed

  • Moved from my forked release of pymp4 (rlaphoenix-pymp4) back to the original pymp4 release as it is
    now up-to-date with some of my needed fixes.
  • The DASH manifest is now stored in the Track url property to be reused by DASH.download_track().
  • Encrypted DASH streams are now downloaded in full and then decrypted, instead of downloading and decrypting
    each individual segment. Unlike HLS, DASH cannot dynamically switch out the DRM/Protection information.
    This brings both CPU and Disk IOPS improvements, as well as fixing rare weird decryption anomalies like broken
    or odd timestamps, decryption failures, or broken a/v continuity.
  • When a track is being decrypted, it now displays "Decrypting" and afterward "Decrypted" in place of the download
    speed.
  • When a track finishes downloaded, it now displays "Downloaded" in place of the download speed.
  • When licensing is needed and fails, the track will display "FAILED" in place of the download speed. The track
    download will cancel and all other track downloads will be skipped/cancelled; downloading will end.
  • The fancy smart quotes ( and ) are now stripped from filenames.
  • All available services are now listed if you provide an invalid service tag/alias.
  • If a WVD file fails to load and looks to be in the older unsupported v1 format, then instructions on migrating to
    v2 will be displayed.
  • If Shaka-Packager prints an error (i.e., :ERROR: log message) it will now raise a subprocess.CalledProcessError
    exception, even if the process return code is 0.
  • The Video classes' Primaries, Transfer, and Matrix classes had changes to their enum names to better represent their
    values and uses. See the changed names in the commit.
  • SubRip (SRT) Subtitles no longer have the MULTI-LANGUAGE SRT header forcefully removed. The root cause of the error
    was identified and fixed in this release.
  • Since Range.Transfer.SDR_BT_601_625 = 5 has been removed, Range.from_cicp() now internally remaps CICP transfer
    values of 5 to 6 (which is now Range.Transfer.BT_601 = 6).
  • Referer and User-Agent Header values passed to the aria2(c) downloader is now set via the dedicated --referer and
    --user-agent options respectively, instead of --header.
  • The aria2(c) -j, -x, and -s option values can now be set by the config under the aria2c key in the options'
    full names.
  • The aria2(c) -x, and -s option values now use aria2(c)'s own default values for them instead of 16. The j
    option value defaults to ThreadPoolExecutor's algorithm of min(32,(cpu_count+4)).
  • The download progress bar now states LICENSING on the speed text when licensing DRM, and LICENSED once finished.
  • The download progress bar now states CANCELLING/CANCELLED on the speed text when cancelling downloads. This is to
    make it more clear that it didn't just stop, but stopped as it was cancelled.
  • The download cancel/skip events were moved to constants.py so it can be used across the codebase easier without
    argument drilling. DL_POOL_STOP was renamed to DOWNLOAD_CANCELLED and DL_POOL_SKIP to DOWNLOAD_LICENCE_ONLY.
  • The Cookie header is now calculated for each URL passed to the aria2(c) downloader based on the URL. Instead of
    passing every single cookie, which could have two cookies with the same name aimed for different host names, we now
    pass only cookies intended for the URL.
  • The aria2(c) process no longer prints output to the terminal directly. Devine now only prints contents of the
    captured log messages to the terminal. This allows filtering out of errors and warnings that isn't a problem.
  • DASH and HLS no longer download segments silencing errors on all but the last retry as the downloader rework makes
    this unnecessary. The errors will only be printed on the final retry regardless.
  • Track.repackage() now saves as {name}_repack.{ext} instead of {name}.repack.{ext}.
  • Video.change_color_range() now saves as {name}_{limited|full}_range.{ext} instead of {name}.range{0|1}.{ext}.
  • Widevine.decrypt() now saves as {name}_decrypted.{ext} instead of {name}.decrypted.{ext}.
  • Files starting with the save path's name and using the save path's extension, but not the save path, are no longer
    deleted on download finish/stop/failure.
  • The output container format is now explicitly specified as MP4 when calling shaka-packager.
  • The default downloader is now requests instead of aria2c to reduce required external dependencies.
  • Reworked the Chapter class to only hold a timestamp and name value with an ID automatically generated as a CRC32 of
    the Chapter representation.
  • The --group option has been renamed to --tag.
  • The config file is now read from three more locations in the following order:
    1. The Devine Namespace Folder (e.g., %appdata%/Python/Python311/site-packages/devine/devine.yaml).
    2. The Parent Folder to the Devine Namespace Folder (e.g., %appdata%/Python/Python311/site-packages/devine.yaml).
    3. The AppDirs User Config Folder (e.g., %localappdata%/devine/devine.yaml).
      Location 2 allows having a config at the root of a portable folder.
  • An empty config file is no longer created when no config file is found.
  • You can now set a default cookie file for a Service, see README.
  • You can now set a default credential for a Service, see config.
  • Services are now auth-less by default and the error for not having at least a cookie or credential is removed.
    Cookies/Credentials will only be loaded if a default one for the service is available, or if you use -p/--profile
    and the profile exists.
  • Subtitles when converting to SubRip (SRT) via SubtitleEdit will now use the /ConvertColorsToDialog option.
  • HLS segments are now merged by discontinuity instead of all at once. The merged discontinuities are then finally
    merged to one file using ffmpeg. Doing the final merge by byte concatenation did not work for some playlists.
  • The Track is no longer passed through Event Callables. If you are able to set a function on an Even Callable, then
    you should have access to the track reference to call it directly if needed.
  • The Track.OnDecrypted event callable is now passed the D...
Read more

v2.2.0

23 Apr 17:40
Compare
Choose a tag to compare

Breaking Changes

Since -q/--quality has been reworked to support specifying multiple qualities, the type of this value is
no longer None|int. It is now list[int] and the list may be empty. It is no longer ever a None value.

Please make sure any Service code that uses quality via ctx.parent.params reflects this change. You may
need to go from an if quality: ... to for res in quality: ..., or such. You may still use if quality
to check if it has 1 or more resolution specified, but make sure that the code within that if tree supports
more than 1 value in the quality variable, which is now a list. Note that the list will always be in
descending order regardless of how the user specified them.

Added

  • Added the ability to specify and download multiple resolutions with -q/--quality. E.g., -q 1080p,720p.
  • Added support for DASH manifests that use SegmentList with range values on the Initialization definition (#47).
  • Added a check for uuid mp4 boxes containing tenc box data when getting the Track's Key ID to improve
    chances of finding a Key ID.

Changed

  • The download path is no longer printed after each download. The simple reason is it felt unnecessary.
    It filled up a fair amount of vertical space for information you should already know.
  • The logs after a download finishes has been split into two logs. One after the actual downloading process
    and the other after the multiplexing process. The downloading process has its own timer as well, so you can
    see how long the downloads itself took.
  • I've switched from using the official pymp4 (for now) with my fork. At the time this change was made the
    original bearypig pymp4 repo was stagnant and the PyPI releases were old. I forked it, added some fixes
    by TrueDread and released my own update to PyPI, so it's no longer outdated. This was needed for some
    mp4 box parsing fixes. Since then the original repo is no longer stagnant, and a new release was made on
    PyPI. However, my repo still has some of TrueDread's fixes that is not yet on the original repository nor
    on PyPI.

Removed

  • Removed the with_resolution method in the Tracks class. It has been replaced with by_resolutions. The
    new replacement method supports getting all or n amount of tracks by resolution instead of the original
    always getting all tracks by resolution.
  • Removed the select_per_language method in the Tracks class. It has been replaced with by_language. The
    new replacement method supports getting all or n amount of tracks by language instead of the original only
    able to get one track by language. It now defaults to getting all tracks by language.

Fixed

  • Prevented some duplicate Widevine tree logs under specific edge-cases.
  • The Subtitle parse method no longer absorbs the syntax error message.
  • Replaced all negative size values with 0 on TTML subtitles as a negative value would cause syntax errors.
  • Fixed crash during decryption when shaka-packager skips decryption of a segment as it had no actual data and
    was just headers.
  • Fixed CCExtractor crash in some scenarios by repacking the video stream prior to extraction.
  • Fixed rare crash when calculating download speed of DASH and HLS downloads where a segment immediately finished
    after the previous segment. This seemed to only happen on the very last segment in rare situations.
  • Fixed some failures parsing tenc mp4 boxes when obtaining the track's Key ID by using my own fork of pymp4
    with up-to-date code and further fixes.
  • Fixed crashes when parsing some tenc mp4 boxes by simply skipping tenc boxes that fail to parse. This happens
    because some services seem to mix up the data of the tenc box with that of another type of box.
  • Fixed using invalid tenc boxes by skipping ones with a version number greater than 1.

v2.1.0

16 Mar 20:53
Compare
Choose a tag to compare

Added

  • The Track get_init_segment method has been re-written to be more controllable. A specific Byte-range, URL, and
    maximum size can now be specified. A manually specified URL will override the Track's current URL. The Byte-range
    will override the fallback value of 0-20000 (where 20000 is the default maximum_size). It now also checks if the
    server supports Byte-range, or it will otherwise stream the response. It also tries to get the file size length and
    uses that instead of maximum_size unless it's bigger than maximum_size.
  • Added new get_key_id method to Track to probe the track for a track-specific Encryption Key ID. This is similar to
    Widevine's from_track method but ignores all pssh boxes and manifest information as the information within those
    could be for a wider range of tracks or not for that track at all.
  • Added a 5-attempt retry system to DASH and HLS downloads. URL downloads only uses aria2(c)'s built in retry system
    which has the same amount of tries and same delay between attempts. Any errors emitted when downloading segments will
    not be printed to console unless it occurred on the last attempt.
  • Added a fallback way to obtain language information by taking it from the representation ID value, which may have the
    language code within it. E.g., audio_en=128000 would be an English audio track at 128kb/s. We now take the en
    from that ID where possible.
  • Added support for 13-char JS-style timestamp values to the Cacher system.
  • Improved Forced Subtitle recognition by checking for both forced-subtitle and forced_subtitle (#43).

Changed

  • The * symbol is no longer spaced after the Widevine KID:KEY when denoting that it is for this specific PSSH.
    This reduces wasted vertical space.
  • The "aria2 will resume download if the transfer is restarted" logs that occur when aria2(c) handles the CTRL+C break,
    and "If there are any errors, then see the log file" logs are now ignored and no longer logged to the console.
  • DASH tracks will no longer prepare and license DRM unless it's just about to download. This is to reduce unnecessary
    preparation of DRM if the track had been converted to a URL download.
  • For a fix listed below, we now use a fork of https://github.com/globocom/m3u8 that fixes a glaring problem with the
    EXT-X-KEY parsing system. See globocom/m3u8#313.
  • The return code when mkvmerge returns an error is now logged with the error message.
  • SubtitleEdit has been silenced when using it for SDH stripping.

Fixed

  • Fixed URL joining and Base URL calculations on DASH manifests that use multiple Base URL values.
  • URL downloads will now store the chosen DRM before preparing and licensing with the DRM.
  • URL downloads will now prepare and license with the DRM if the Track has pre-existing DRM information. Previously it
    would only prepare and license DRM if it did not pre-emptively have DRM information before downloading.
  • The * symbol that indicates that the KID:KEY is for the track being downloaded now uses the new get_key_id method
    of the track for a more accurate reading.
  • License check now ensures if a KEY was returned for the Track instead of all KIDs of the Track's PSSH. This prevents
    an issue where the PSSH may have Key IDs for a 720p and 1080p track, yet only a KEY for the 720p track was returned.
    It would have then raised an error and stopped the download, even though you are downloading the 720p track and not
    the 1080p track, therefore the error was irrelevant.
  • Unnecessary duplicate license calls are now prevented in some scenarios where --cdm-only is used.
  • Fixed accuracy and speed of preparing and licensing DRM on HLS manifests where multiple EXT-X-KEY definitions appear
    in the manifest throughout the file. Using globocom/m3u8#313 we can now accurately get a
    list of EXT-X-KEYs mapped to each segment. This is a game changer for HLS manifests that use unique keys for every
    single (or most) segments as it would have otherwised needed to initialize (and possibly do network requests) for
    100s of EXT-X-KEY information, per segment. This caused downloads of HLS manifests that used a unique key per segment
    to slow to a binding crawl, and still not even decrypt correctly as it wouldn't be able to map the correct initialized
    key to the correct segment.
  • Fixed a regression that incorrectly implemented the OnMultiplex event for Audio and Subtitle tracks causing them to
    never trigger. It would instead accidentally have trigger the last Video track's OnMultiplex event instead of the
    Audio or Subtitle's event.
  • The above fix also fixed the automatic SDH stripping subtitle. Any automatically created SDH->non-SDH subtitle from
    prior downloads would not have actually had SDH captions stripped, it would instead be a duplicate subtitle.

New Contributors

v2.0.1

07 Mar 10:29
Compare
Choose a tag to compare

Added

  • Re-added logging support for shaka-packager on errors and warnings. Do note that INFO logs and the 'Insufficient bits
    in bitstream for given AVC profile' warning logs are ignored and never printed.
  • Added new exceptions to the Widevine DRM class, CEKNotFound and EmptyLicense.
  • Added support for Byte-ranges on HLS init maps.

Changed

  • Now lists the full 'Episode #' text when listing episode titles without an episode name.
  • Subprocess exceptions from a download worker no longer prints a traceback. It now only logs the return code. This is
    because all subprocess errors during a download is now logged, therefore the full traceback is no longer necessary.
  • Aria2(c) no longer pre-allocates file space if segmented. This is to reduce generally unnecessary upfront I/O usage.
  • The Widevine DRM class's get_content_keys method now raises the new CEKNotFound and EmptyLicense exceptions not
    ValueError exceptions.
  • The prepare_drm code now raises exceptions where needed instead of sys.exit(1). Callees do not need to make any
    changes. The exception should continue to go up the call stack and get handled by the dl command.

Fixed

  • Fixed regression that broke support for pproxy. Do note that while pproxy has wheel's for Python 3.11+, it seems to
    be broken. I recommend using Python 3.10 or older for now. See qwj/python-proxy#161.
  • Fixed regression and now store the chosen DRM object back to the track.drm field. Please note that using the track
    DRM field in Service code is not recommended, but for some services it's simply required.
  • Fixed regression since v1.4.0 where the byte-range calculation was actually slightly off one on the right-side range.
    This was a one-indexed vs. zero-indexed problem. Please note that this could have affected the integrity of HLS
    downloads if they used EXT-X-BYTERANGE.
  • Fixed possible soft-lock in HLS if the Queue for previous segment key and init data gets stuck in an empty state over
    an exception in a download thread. E.g., if a thread takes the previous segment key, throws an exception, and did not
    get the chance to give it back for the next thread.
  • The prepare_drm function now handles unexpected exceptions raised in the Service's license method. This code would of
    otherwise been absorbed and the download would have soft-locked.
  • Prevented a double-licensing call race-condition on HLS tracks by using a threading lock when preparing DRM
    information. This is not required in DASH, as it prepares DRM on the main thread, once, not per-segment.
  • Fixed printing of aria2(c) logs when redirecting progress information to rich progress bars.
  • Explicitly mark DASH and HLS aria2(c) downloads as segmented.
  • Fixed listing of episode titles without an episode name.
  • Fixed centering of the project URL in the ASCII banner.
  • Removed the accidental double-newline after the ASCII banner.