Version 1.3

Bundled all functionality into one executable

Previously there were different executables for different actions. This model of bundling actions conflicts with a
static
build of isisdl, since you would have to build every executable separately.

Added external links

Now the entire course page is searched for anything that matches an url (regex based). If one is found, it gets queried
and
if the Content-Type is application/*, the page is downloaded.

This means that any url mentioned in ISIS, will now appear as part of the course. Since a web-request has to be sent for
every external url, it takes quite a lot of time. The downloaded information is of course cached, so it does not have to
be retrieved every time.

Note that the TU Berlin is liable for content posted on ISIS, but not for external links. So if you do not want this
risk,
you might want to consider disabling this option.

More configuration options

Now added the following options:

• Renaming courses
• Following external links
• Should subdirectories be created in the main course directory?
• Should file names be displayed as absolute paths?

If you open a not downloaded file, it will get downloaded with priority

This is made possible by the Linux Kernel API inotify. Whenever any program opens any file in the ~/isisdl directory,
the streaming thread will get notified. It pauses all other downloads and downloads the opened file with maximum
bandwidth.

This is especially useful, if you either don't want to download all video files, but still be able to view them
with vlc, or if you want to start watching videos while the download is still running.

Specifically for the former use case I added an option --stream, which will launch only the streaming
thread.

Subscribe to all courses

Currently unsupported due to legal reasons

In order for me to limit test isisdl, I want to have access to as many courses as possible. Because of that I decided
to implement a way of subscribing to all publicly available courses.

It is done by trying to subscribe to every course with the API. If it was successful, the result is stored in the
database. With this it is also possible to unsubscribe from all courses automatically. Unfortunately this is quite slow
since unsubscribing via the Moodle API is not supported… (yet).

Minor changes

Better progress bar. Split phases in two so the status is actually accurate.

Changed the master password to a sha digest of a string. It now has 64 characters.

Before launching it is checked if you have an active internet connection. If not, isisdl is exited gracefully.

isisdl will now only ever prompt you 1x for your password.

Fixed a bug where, if you throttle too high (2MiB/s), nothing would get downloaded.

A few bug fixes here and there. Probably also introduced some more bugs :D

Version 1.1

isisdl version 1.1

A new major version is here! 🥳 🎉

Highlights:

• isisdl-compress: Compress your videos into the H265 codec
• Systemd timers
• configuration file [linux exclusive]
• New (experimental) installation method

Breaking changes

In order to account for the breaking changes I have introduced, the directory name has changed. Now it is called isisdl, located in your home directory. All courses are downloaded to there without the previous directory Courses/. Also, there is no .intern/ directory anymore, only the .state.db Database.

This should be the last time I rename this directory. Let's see if it will be that way!

Detailed Changelog

ffmpeg compression to H265

When executing isisdl-compress, ffmpeg is launched with options to compress your files (only for the courses you whitelist). These are sorted by maximum bitrate such that the files with the highest expected efficiency are compressed first. Thus, the efficiency will start out very high and drop gradually. It should be a smooth curve and the compression can be stopped at the point where you think compression is not worth it anymore. I set this point at 50%.

The compression is very efficient. I could compress the first 20 GiB into just 3 GiB. Of course over time this will drop down, but for the first few hours it is definitely worth it.

The compressed videos are also nicely integrated into isisdl. Compressed files are not re-downloaded if either of the following is true

• You don't delete / lose the ~/isisdl/.state.db File
• You don't rename compressed files

If you delete ~/isisdl/.state.db and rename the compressed files it still is not that big of a deal since they won't be overwritten (they have a different file name). The isisdl-sync prompt will, however, notify you of unrecognized files.

In version 1.2 I will use the mp4 metadata to deduce the original filename and this problem won't exist anymore.

Status

There is a detailed status info on

• Total size before / after compression
• Global efficiency
• How much space was saved
• How long the current file will take
• Estimated current file size

Quitting while compressing

Sending a quit-signal to isisdl while in the middle of compressing is supported - the current compression will finish and an overview of the courses and their file-size-savings is displayed. However, if you quit isisdl while in the phase of waiting for the last compression to finish, ffmpeg will produce corrupted output. Since the output is saved to a temporary file first it is not a problem. Beware that you lose the progress that you have made for that file.

Side note: global time

It is not possible to calculate the global time since that would require starting every compression and then estimating the total time based on how much it took e.g. for 10s. However, if ffmpeg is interrupted during compression it can't be re-launched. And compression 10s of every video to throw it away again is horribly inefficient.

New (experimental) installation method

I have discovered the python module nuitka, which allows for compiling python code directly to native executables. These are huge: I am taking 20 MiB for a single one. This is due to the python runtime being insanely large. It is, however, acceptable since it provides great convenience. You don't have do install isisdl from pip but rather only download the executable(s) you need.

In theory this allows you to execute isisdl without having python installed. But don't quote me on that. I will test the compatibility and static-ness of the executables in the near future. For now it is an experimental installation method.

Unencrypted passwords are not stored unencrypted anymore

Previously if you selected to store the password unencrypted they were saved "as is" in the database. This in not really that problematic - since your computer should be free from malware - however it is better this way. This way of handling passwords is inspired from Google Chrome.

New configuration file

If you are running a Unix-like environment, you now have an example config located at ~/.config/isisdl/example.yaml once you execute isisdl for the first time. Here you can set every option that is available. You can do things like change the name of main working directory, database file name and other things you could set via the wizard.

In fact this file can also set options that you could not set with the wizard. All options which you set are interpreted by the settings.py file, which has all the static configuration. Here you could set things like the download_chunk_size, status_time or master_password to encrypt if no password was provided.

Usually you don't want to mess with these settings since they are chosen in the best way I could find. If you do find a setting that benefits from a different value do let me know.

Systemd-timers

Systemd-timers provide a huge feature set and are "better" than Cronjobs. See this and this link for further details. While I don't use the most part of the features it provides at least the dependency of cron is now eliminated. Of course, you will have to have a systemd-based distribution to take advantage of them. But who hasn't these days?

The location of the files is ~/.config/systemd/user/isisdl.timer while the service is located at ~/.config/systemd/user/isisdl.service. This timer is set to launch isisdl every hour, however it can be set to whatever time you feel is appropriate for your configuration.

Under the hood: config

I used to have a table in the Database responsible for configuration. A new value would be INSERT-ed into it. This way of doing it was syntactically very noisy and had a few own issues. Now there exists a class that takes care of it. Here attributes can be type-annotated and defined with __slots__ removing the functionality of arbitrarily setting attributes. A small caveat following from this is that json-strings are stored in the database. I've seen a meme about it, but I think this is the most convenient way to store it.

Also, the config values now have readable names instead of "0", "1", "2", …

Minor details

Better sanitation of filenames

Since the directory name changed the absolute path of every downloaded file now has no special characters in it. Also removed the option to only replace some characters - now it is all or nothing.

New update policy

Now isisdl has the ability to notify you about an update without actually installing it.

Better exception handling

All blocks that may fail are wrapped in a big

try:
    ...
except Exception:
    generate_error_message()

block. When an error occurs it is logged to a file and the program is exited gracefully.

Version 1.0

This release marks the first official release of isisdl.

Updates:

Major backend change

isisdl now uses the API backend of ISIS instead of web-Scraping the response from the web-server. With this feature it is now possible to collect all needed information in O(#courses) instead of O(#files). This is a major improvement for runtime as I've previously waited ~1min for the downloads to start which now decreased to ~5s. This also removes Beautifulsoup from the requirements.

With this change isisdl does not have to use the stupid download_folder.php form to download a folder of Files. Instead all files may be accessed directly.

Another minor benefit is that now the urls which get retrieved are urls to the direct file location instead of a redirect. This also improves runtime slightly.

Minor frontend change

The frontend - or at least the design I'm going for - did not change much.

One thing, that did change however is a more efficient implementation of redrawing the status. Previously I've relied on the clear / cls command to clear the screen and re-print everything. This is horribly inefficient. I just didn't know a better way.
Recently I've discovered that there are terminal escape sequences to move the cursor up and down. This is used now.

Future ideas

For the most part I don't have planned much more for this library. It does what is should and does so very well.

Auto-compression of videos

The only feature that is currently missing is auto-compression of videos to the H.265 codec. This is what I usually do before a semester ends such that the videos I save won't take up that much space (and yes, the tradeoff between cpu time and space might not be worth). To automate and distribute the work would be greatly beneficial for everybody.

Sharing these videos

I have access to a server which may be set up to host the compressed videos. These can be automatically retrieved and downloaded instead of their H.264 counterpart. It will happen transparently and you only save file size.

Unfortunately I will have to restrict access to this server since

• It has a data cap
• I must ensure that no copyright infringement is happening.

Also the Server only has 15GB storage space. So it will be limited in storage space as well.

Version 0.4

Added a bunch of features which are way to many to list.
Everything is better now!