Compression

This page describes how libdragon achieves data compression in general for different type of assets (executables, graphics, audio). It gives some general guidance and explains the rationale of the various defaults.

Overview of compression in libdragon

Libdragon ships with three different data compression algorithms, generically identified by level numbers from 1 to 3 (with "level 0" sometimes being used to refer to "no compression"). Increasing the level means slower compression during builds, slower decompressions at runtime, but more compression ratio.

The actual algorithm behind each level is considered an implementation detail; as new algorithms are tested on N64, libdragon can change the algorithms at any point, even in the stable branch (that is, a change of compression algorithm is not considered a breaking change). It is thus important that compression of data happens during build, and compressed data is not committed to the game repository: such pre-compressed, committed data might in fact turn out to be unusable with future libdragon versions.

• Level 1: this is fastest compression level. The ratio is not great (normally worse than gzip, just to give a ballpark) but it is so fast at decompressing that it achieves an important property: data compressed at level 1 is faster at loading and decompressing, than loading uncompressed data. This surprising result happens because the time saved by loading data fewer bytes from ROM is enough to counterbalance the time spent in decompressing the data. This property has been proven true with so many different kind of data, that can be considered true for all data. Currently, the algorithm used is LZ4 by Yann Collet.
• Level 2: this is a good intermediate compression level. The ratio is quite good (better than gzip -9 in most cases), but it is still decently fast at decompressing. Currently, the algorithm uses is LHA-LH5 in the stable branch, and Aplib in the preview branch.
• Level 3: this is an ultra high compression level. It is tuned for absolutely maximum compression at the expense of decompression time. The compression ratio is incredibly high, and it approaches best-in-class on PC too (like xz or 7zip), thanks also to the fact that N64 files are generally quite small by today's standard. Decompression time is quite high though: turning it on for all assets will definitely, noticeably increase loading times for your game, and possibly making them too long. It is mostly meant to be used for rarely loaded assets (eg: error screens) or in situations where loading times are not important. It is currently available only in the preview branch, and it uses the Shrinkler algorithm

This is a table that recaps some datapoints (measured against a fixed corpus of game assets) and give some advice.

Level	Ratio	Ratio vs gzip	Load+Decomp speed	Notes
1	20%	+2%	100 KiB/s	Faster than uncompressed, used by default on everything
2	30%	+10%	200 KiB/s	Good for assets loaded during level changes; might not be fast enough for data streamed during game
3	50%	+40%	30 KiB/s	Loading is very slow; good for assets loaded very rarely like error screen, or for situations where loading time is not a huge issue

Compressing and loading images (sprites, textures) and fonts

Normally, images are processed via the mksprite tool and converted into the .sprite format. Fonts instead are processed via the mkfont tool and converted into the .font64 format.

Both of these formats accept the command line option -c or --compress which allows to specify the compression level to be used for these files. By default, level 1 is used. To change the compression level, you can add the option to your Makefile like this:

filesystem/%.sprite: assets/%.png
	@mkdir -p $(dir $@)
	@echo "    [SPRITE] $@"
	@$(N64_MKSPRITE) --compress 2 -o "$(dir $@)" "$<"

Note that libdragon does not support any form of lossy image compression at the moment (es: jpeg).

Compressing arbitrary data files

Libdragon compression library can be used also to compress your own arbitrary files. To do so, you must use the mkasset tool to compress the files, and then load them using the asset library (asset.h).

Invoking mkasset manually is quite easy:

$ $N64_INST/bin/mkasset --compression 1 -o filesystem myfile.dat

By default, the file will be overwritten with the compressed version, or you can use -o to specify an output directory. You can even pass multiple files on the command line if you want.

Invoking mkasset from a Makefile can be done with a custom rule:

filesystem/%.dat: assets/%.dat
	@mkdir -p $(dir $@)
	@echo "    [DAT] $@"
	@$(N64_BINDIR)/mkasset --compress 2 -o "$(dir $@)" "$<"

The above rule will compress via mkasset all .dat files in the assets/ directory, and put the compressed files into the filesystem directory.

Before loading the assets, you must initialize the compression algorithms. This is required so that libdragon can avoid linking in too much code into the final game: if a certain compression algorithm is not used by your assets, it will no the linked. To initalize each compression algorithm, use asset_init_compression:

// Initialize compression level 3
asset_init_compression(3);

Notice that compression level 1 is initialized by default, so you don't need to do anything if you use the default compression level.

The asset library provides two main entrypoints: asset_load() and asset_fopen().

asset_load() just loads the whole file into memory, uncompresses it and return a pointer to it:

int sz;
void *data = asset_load("rom:/mydata.dat", &sz);

It also returns the (uncompressed) size of the asset in case it is useful. The returned buffer can be disposed via free().

asset_fopen() can be used to parse the file as it is being read (and uncompressed) from disk, without reading it all into RAM:

int sz;
FILE *f = asset_fopen("rom:/mydata.dat", &sz);

char id[3];
fread(id, 1, 3, f); 
if (memcmp(id, "ID", 3) != 0) {
   ....
}

[ ... proceed reading the file ... ]

Both asset_load and asset_fopen also works on uncompressed files, not processed via mkasset. This is made it so you can start using them from day zero has a way to load your data into the game, and then later on add compression if needed.

Compressing game code

Libdragon bootcode (IPL3) supports decompression of game code at boot time. Libdragon makefiles using n64.mk are already setup for that: by default, it compresses the game code with level 1 compression, which makes the ROM actually boots faster than uncompressed.

To change the compression level of game code, simply adds this variable to your Makefile:

# Set game code compression to level 2
N64_ROM_ELFCOMPRESS = 2

DSOs (overlays, that is dynamic libraries of code that is loaded at runtime) are also compressed by default with level 1. To change that, a separate Makefile variable is used:

# Set DSOs compression to level 2
DSO_COMPRESS_LEVEL = 2

This is an example testing the various compression levels on the "Brew Volley" game example in libdragon:

Level	Size	Ratio	Compression time
0	412 KiB	100.0%	0.00 s
1	245 KiB	59.5%	0.39 s
2	177 KiB	43.0%	2.93 s
3	155 KiB	37.6%	6.09 s

Compression times were measured on an Apple MacBook Pro 2021 with M1 Pro. Notice that compression times do impact the development cycle when activated in the Makefile, as the time is paid every time a new ROM is made. It is advised to keep compression at level 1 for the standard development cycle.

For comparison, gzip --best produces a 188 KiB file, and xz --best produces a 150 KiB file.

Note

Compression of game code is only supported in the preview branch at the moment.