Add IO to OpenEducation methodology

Signed-off-by: Mihnea Firoiu <[email protected]>

chapters/io/Makefile

@@ -0,0 +1,33 @@
+RVMD = reveal-md
+MDPP = markdown-pp
+FFMPEG = ffmpeg
+
+SLIDES ?= slides.mdpp
+SLIDES_OUT ?= slides.md
+MEDIA_DIR ?= media
+SITE ?= _site
+OPEN ?= xdg-open
+
+.PHONY: all html clean videos
+
+all: videos html
+
+html: $(SITE)
+
+$(SITE): $(SLIDES)
+	$(MDPP) $< -o $(SLIDES_OUT)
+	$(RVMD) $(SLIDES_OUT) --static $@
+
+videos:
+	for TARGET in $(TARGETS); do \
+		$(FFMPEG) -framerate 0.5 -f image2 -y \
+			-i "$(MEDIA_DIR)/$$TARGET/$$TARGET-%d.svg" -vf format=yuv420p $(MEDIA_DIR)/$$TARGET-generated.gif; \
+	done
+
+open: $(SITE)
+	$(OPEN) $</index.html
+
+clean:
+	-rm -f $(MEDIA_DIR)/*-generated.gif
+	-rm -f *~
+	-rm -fr $(SITE) $(SLIDES_OUT)

content/chapters/io/lab/content/arena.md → chapters/io/arena/drills/tasks/arena.md

@@ -58,7 +58,7 @@ struct file_operations {
 OK, there will be no Blackjack...
 for now at least.
 But there **will** be pipes.
-Navigate back to `support/mini-shell/mini_shell.c` and add support for piping 2 commands together like this:
+Navigate back to `mini-shell/support/mini_shell.c` and add support for piping 2 commands together like this:
 
 ```console
 > cat bosses.txt | head -n 5
@@ -71,7 +71,7 @@ Ancient Dragon
 
 ## To Drop or Not to Drop?
 
-Remember `support/buffering/benchmark_buffering.sh` or `support/file-mappings/benchmark_cp.sh`.
+Remember `buffering/support/benchmark_buffering.sh` or `file-mappings/support/benchmark_cp.sh`.
 They both used this line:
 
 ```bash
@@ -96,12 +96,12 @@ This makes I/O faster.
 The line from which this discussion started invalidates those caches and forces the OS to perform I/O operations "the slow way" by interrogating the disk.
 The scripts use it to benchmark only the C code, not the OS.
 
-To see just how much faster this type of caching is, navigate to `support/buffering/benchmark_buffering.sh` once again and comment-out the line with `sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"`.
+To see just how much faster this type of caching is, navigate to `buffering/support/benchmark_buffering.sh` once again and comment-out the line with `sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"`.
 Now run the script **a few times** and compare the results.
 You should see some drastic improvements in the running times, such as:
 
 ```console
-student@os:/.../support/file-mappings$ ./benchmark_cp.sh
+student@os:/.../file-mappings/support$ ./benchmark_cp.sh
 make: Nothing to be done for 'all'.
 Benchmarking mmap_cp on a 1 GB file...
 
@@ -115,7 +115,7 @@ user    0m0,013s
 sys     0m1,301s
 
 
-student@os:/.../support/file-mappings$ ./benchmark_cp.sh
+student@os:/.../file-mappings/support$ ./benchmark_cp.sh
 make: Nothing to be done for 'all'.
 Benchmarking mmap_cp on a 1 GB file...
 

chapters/io/arena/reading/arena.md

@@ -0,0 +1,144 @@
+# Arena
+
+## Open File Structure in the Kernel
+
+The "open file" `struct` in the Linux kernel is called [`struct file`](https://elixir.bootlin.com/linux/v6.0.9/source/include/linux/fs.h#L940)
+Its most important fields are:
+
+```c
+struct file {
+        struct path                     f_path;
+        /* Identifier within the filesystem. */
+        struct inode                    *f_inode;
+
+        /**
+         * Contains pointers to functions that implement operations that
+         * correspond to syscalls, such as `read()`, `write()`, `lseek()` etc.
+         */
+        const struct file_operations    *f_op;
+
+        /**
+         * Reference count. A `struct file` is deallocated when this reaches 0,
+         * i.e. nobody uses it anymore.
+         */
+        atomic_long_t                   f_count;
+
+        /* Those passed to `open()`. */
+        unsigned int                    f_flags;
+        fmode_t                         f_mode;
+
+        /* Cursor from where reads/writes are made */
+        loff_t                          f_pos;
+        /* To allow `f_pos` to be modified atomically. */
+        struct mutex                    f_pos_lock;
+}
+```
+
+As mentioned above, [`struct file_operations`](https://elixir.bootlin.com/linux/v6.0.9/source/include/linux/fs.h#L2093) contains function pointers that well-known syscalls such as `read()` end up calling.
+Each filesystem needs to define its own implementations of these functions.
+Some of the most widely known `file_operations` are listed below.
+By now, you should recognise most of them:
+
+```c
+struct file_operations {
+        loff_t (*llseek) (struct file *, loff_t, int);
+        ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
+        ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
+        int (*mmap) (struct file *, struct vm_area_struct *);
+        unsigned long mmap_supported_flags;
+        int (*open) (struct inode *, struct file *);
+        int (*flush) (struct file *, fl_owner_t id);
+        int (*fsync) (struct file *, loff_t, loff_t, int datasync);
+        int (*fasync) (int, struct file *, int);
+}
+```
+
+## Mini-shell with Blackjack and Pipes
+
+OK, there will be no Blackjack...
+for now at least.
+But there **will** be pipes.
+Navigate back to `mini-shell/support/mini_shell.c` and add support for piping 2 commands together like this:
+
+```console
+> cat bosses.txt | head -n 5
+Darkeater Midir
+Slave Knight Gael
+Nameless King
+Dancer Of The Boreal Valley
+Ancient Dragon
+```
+
+## To Drop or Not to Drop?
+
+Remember `buffering/support/benchmark_buffering.sh` or `file-mappings/support/benchmark_cp.sh`.
+They both used this line:
+
+```bash
+sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
+```
+
+Note that `sync` has a [man page](https://linux.die.net/man/8/sync) and it partially explains what's going on:
+
+> The kernel keeps data in memory to avoid doing (relatively slow) disk reads and writes. This improves performance
+
+So the kernel does **even more [buffering](./io-internals.md#io-buffering)!**
+But this time, it's not at the syscall level, like with `read()` and `write()`.
+And it's used a bit differently.
+
+While buffering is a means of either receiving data in advance (for reading) or committing it retroactively (for writing) to speed up subsequent syscalls that use the **next data**, caching is a means of speeding up calls that use the **same data**.
+Just like your browser caches the pages you visit so you refresh them faster or your CPU caches your most recently accessed addresses, so does your OS **with your files**.
+
+Some files are read more often than others: logs, some configs etc.
+Upon encountering a first request (read / write) to a file, the kernel stores chunks of them in its memory so that subsequent requests can receive / modify the data in the RAM rather than waiting for the slow disk.
+This makes I/O faster.
+
+The line from which this discussion started invalidates those caches and forces the OS to perform I/O operations "the slow way" by interrogating the disk.
+The scripts use it to benchmark only the C code, not the OS.
+
+To see just how much faster this type of caching is, navigate to `buffering/support/benchmark_buffering.sh` once again and comment-out the line with `sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"`.
+Now run the script **a few times** and compare the results.
+You should see some drastic improvements in the running times, such as:
+
+```console
+student@os:/.../file-mappings/support$ ./benchmark_cp.sh
+make: Nothing to be done for 'all'.
+Benchmarking mmap_cp on a 1 GB file...
+
+real    0m13,299s
+user    0m0,522s
+sys     0m1,695s
+Benchmarking cp on a 1 GB file...
+
+real    0m10,921s
+user    0m0,013s
+sys     0m1,301s
+
+
+student@os:/.../file-mappings/support$ ./benchmark_cp.sh
+make: Nothing to be done for 'all'.
+Benchmarking mmap_cp on a 1 GB file...
+
+real    0m1,286s
+user    0m0,174s
+sys     0m0,763s
+Benchmarking cp on a 1 GB file...
+
+real    0m5,411s
+user    0m0,012s
+sys     0m1,201s
+```
+
+Each subsequent benchmark actually reads the data from the caches populated or refreshed by the previous one.
+
+You can use `free -h` to view how much data your kernel is caching.
+Look at the `buff/cache` column.
+One possible output is shown below.
+It says the OS is caching 7 GB of data.
+
+```console
+student@os:~$ free -h
+              total        used        free      shared  buff/cache   available
+Mem:           15Gi       8,1Gi       503Mi       691Mi       7,0Gi       6,5Gi
+Swap:         7,6Gi       234Mi       7,4Gi
+```

content/chapters/io/lab/content/async-io.md → chapters/io/async-io/reading/async-io.md

@@ -29,7 +29,7 @@ In case of asynchronous I/O, the "backend" used to implement the operations may
 
 ## Practice
 
-Enter the `support/async/` folder for some implementations of a simple request-reply server in Python or in C.
+Enter the `async/support/` folder for some implementations of a simple request-reply server in Python or in C.
 The server gets requests and serves them in different ways: synchronous, multiprocess-based, multi-threading-based, asynchronous.
 
 We use two implementations, in Python and in C.
@@ -57,19 +57,19 @@ We use two implementations, in Python and in C.
    To start the server, run each of these commands (one at a time to test the respective server type):
 
    ```console
-   student@os:/.../support/async/python$ ./server.py 2999
+   student@os:/.../async/support/python$ ./server.py 2999
 
-   student@os:/.../support/async/python$ ./mp_server.py 2999
+   student@os:/.../async/support/python$ ./mp_server.py 2999
 
-   student@os:/.../support/async/python$ ./mt_server.py 2999
+   student@os:/.../async/support/python$ ./mt_server.py 2999
 
-   student@os:/.../support/async/python$ ./async_server_3.6.py 2999
+   student@os:/.../async/support/python$ ./async_server_3.6.py 2999
    ```
 
    For each server, in a different console, we can test to see how well it behaves by running:
 
    ```console
-   student@os:/.../support/async/python$ time ./client_bench.sh
+   student@os:/.../async/support/python$ time ./client_bench.sh
    ```
 
    You will see a time duration difference between `mp_server.py` and the others, `mp_server.py` runs requests faster.
@@ -96,17 +96,17 @@ We use two implementations, in Python and in C.
    Same as with Python, to start the server, run each of these commands (one at a time to test the respective server type):
 
    ```console
-   student@os:/.../support/async/c$ ./server 2999
+   student@os:/.../async/support/c$ ./server 2999
 
-   student@os:/.../support/async/c$ ./mp_server 2999
+   student@os:/.../async/support/c$ ./mp_server 2999
 
-   student@os:/.../support/async/c$ ./mt_server 2999
+   student@os:/.../async/support/c$ ./mt_server 2999
    ```
 
    For each server, in a different console, we can test to see how well it behaves by running:
 
    ```console
-   student@os:/.../support/async/python$ time client_bench.sh
+   student@os:/.../async/support/python$ time client_bench.sh
    ```
 
    We draw 2 conclusions from using the C variant:
@@ -122,7 +122,7 @@ When aiming for performance, asynchronous I/O operations are part of the game.
 And it's very useful having a good understanding of what's happening behind the scenes.
 
 For example, for the Python `async_server_3.6.py` server, a message `asyncio: Using selector: EpollSelector` is provided.
-This means that the backend relies on the use of the [`epoll()` function](https://man7.org/linux/man-pages/man7/epoll.7.html) that's part of the [I/O Multiplexing section](./io-multiplexing.md).
+This means that the backend relies on the use of the [`epoll()` function](https://man7.org/linux/man-pages/man7/epoll.7.html) that's part of the [I/O Multiplexing section](../../io-multiplexing/reading/io-multiplexing.md).
 
 Also, for Python, the use of the GIL may be an issue when the operations are CPU intensive.
 
@@ -131,4 +131,4 @@ It's rare that servers or programs using asynchronous operations are CPU intensi
 It's more likely that they are I/O intensive, and the challenge is avoiding blocking points in multiple I/O channels;
 not avoiding doing a lot of processing, as is the case with the `fibonacci()` function.
 In that particular case, having thread-based asynchronous I/O and the GIL will be a good option, as you rely on the thread scheduler to be able to serve multiple I/O channels simultaneously.
-This later approach is called I/O multiplexing, discussed in [the next section](./io-multiplexing.md).
+This later approach is called I/O multiplexing, discussed in [the next section](../../io-multiplexing/reading/io-multiplexing.md).

content/chapters/io/lab/content/beyond-network-sockets.md → chapters/io/beyond-network-sockets/reading/beyond-network-sockets.md

@@ -1,9 +1,9 @@
 # Beyond Network Sockets
 
-Up until this point, we first learned how to use the [Berkeley Sockets API](./remote-io.md#api---hail-berkeley-sockets), then we learned about the [client-server model](./client-server-model.md), based on this API.
+Up until this point, we first learned how to use the [Berkeley Sockets API](./remote-io.md#api---hail-berkeley-sockets), then we learned about the [client-server model](../../client-server-model/reading/client-server-model.md), based on this API.
 So now we know that sockets offer a ubiquitous interface for inter-process communication, which is great.
 A program written in Python can easily send data to another one written in C, D, Java, Haskell, you name it.
-However, in the [section dedicated to networking](./networking-101.md), we saw that it takes a whole stack of protocols to send this message from one process to the other.
+However, in the [section dedicated to networking](../../networking-101/reading/networking-101.md), we saw that it takes a whole stack of protocols to send this message from one process to the other.
 As you might imagine, this is **much slower even than local I/O using files**.
 
 So far we've only used sockets for local communication, but in practice it is a bit counterproductive to use network sockets for local IPC due to their high latency.
@@ -13,7 +13,7 @@ Well, there is a way and it's called **UNIX sockets**.
 ## UNIX Sockets
 
 UNIX sockets are created using the `socket()` syscall and are bound **TO A FILE** instead of an IP and port using `bind()`.
-You may already see a similarity with [named pipes](./pipes.md#named-pipes---mkfifo).
+You may already see a similarity with [named pipes](../../pipes/reading/pipes.md#named-pipes---mkfifo).
 Just like them, UNIX sockets don't work by writing data to the file (that would be slow), but instead the kernel retains the data they send internally so that `send()` and `recv()` can read it from the kernel's storage.
 You can use `read()` and `write()` to read/write data from/to both network and UNIX sockets as well, by the way.
 The differences between using `send()`/`recv()` or `write()`/`read()` are rather subtle and are described in [this Stack Overflow thread](https://stackoverflow.com/questions/1790750/what-is-the-difference-between-read-and-recv-and-between-send-and-write).
@@ -23,7 +23,7 @@ However, there are [third-party libraries](https://crates.io/crates/uds_windows)
 
 ### Practice: Receive from UNIX Socket
 
-Navigate to `support/receive-challenges/receive_unix_socket.c`.
+Navigate to `receive-challenges/support/receive_unix_socket.c`.
 Don't write any code yet.
 Let's compare UNIX sockets with network sockets first:
 

content/chapters/io/lab/content/client-server-model.md → chapters/io/client-server-model/reading/client-server-model.md

@@ -13,7 +13,7 @@ What's different from UDP is that this connection is **bidirectional**, so we ca
 Notice that the syscalls have changed.
 We were using `sendto()` and `recvfrom()` for UDP, and now we're using `send()` and `recv()` for TCP.
 And yes, despite the fact that we're using Python, these are syscalls.
-You saw them in C when you solved the [challenge](./remote-io.md#practice-network-sockets-challenge).
+You saw them in C when you solved the [challenge](../../remote-io/reading/remote-io.md#practice-network-sockets-challenge).
 
 ## Server vs Client
 
@@ -24,7 +24,7 @@ Either way, it is **listening** for connections.
 
 The client is the active actor, being the one who initiates the connection.
 
-[Quiz](../quiz/client-server-sender-receiver.md)
+[Quiz](../drills/questions/client-server-sender-receiver.md)
 
 ## Establishing the Connection
 
@@ -65,7 +65,7 @@ Below is an image summarising the steps above:
 
 ### Practice: Client
 
-Navigate to `support/client-server/`.
+Navigate to `client-server/support/`.
 Here you will find a minimalistic server implementation in `server.py`.
 
 1. Read the code and identify the steps outlined above.
@@ -77,7 +77,7 @@ Run multiple clients.
 
 ## Practice: Just a Little Bit More Deluge
 
-We've already said that Deluge uses an [abstraction over TCP](./networking-101.md#practice-encapsulation-example-deluge-revived) to handle socket operations, so we don't have the luxury of seeing it perform remote I/O "manually".
+We've already said that Deluge uses an [abstraction over TCP](../../networking-101/reading/networking-101.md#practice-encapsulation-example-deluge-revived) to handle socket operations, so we don't have the luxury of seeing it perform remote I/O "manually".
 However, there are a few instances where Deluge uses socket operations itself, mostly for testing purposes.
 
 Deluge saves its PIDs (it can spawn multiple processes) and ports in a file.