LZW streaming compressor/decompressor

Github Actions	Codecov

LZW streaming compressor/decompressor based on LZW AB compatible with UNIX compress.

Compressor dictionaries

• Linked list (idea from LZW AB). It has low memory usage <= 327 KB (16 bit codes). It is slow in general, recommended for small amount of data.
• Sparse array (enabled by default). It has high memory usage <= 33.5 MB (16 bit codes). It is fast, recommended for large amount of data.

You can add your own implementation, see api for dictionary.

Why?

• You can compress and decompress .Z (tar.Z) archives.
• You can resurrect legacy apis (like Content-Encoding: compress in HTTP 1.1).
• You can support ancient software and make your application looks like ancient software.

New features

• Accurate ratio calculation without overhead, compressor provides smaller archive (compatible with UNIX compress).
• Unaligned bit groups switch (only disabled mode is compatible with UNIX compress).
• LSB/MSB switch (only LSB mode is compatible with UNIX compress).
• Magic header switch (only enabled magic header is compatible with UNIX compress).

Dependencies

Operating systems: GNU/Linux, OSX, FreeBSD, Windows (Visual Studio, MinGW, Cygwin).

• Runtime and compilation dependencies: GMP or TomMath.
• Compilation dependencies: cmake, gcc or clang or msvc, asciidoc, docbook xml 4.5.
• Additional development tools: yq.

Popular OS	Runtime dependencies	Compilation dependencies
Ubuntu	libgmp10/libtommath1	libgmp-dev/libtommath-dev
		cmake, gcc/clang, asciidoc, docbook-xml
CentOS	gmp/libtommath	gmp-devel/libtommath-devel
		cmake3, gcc/clang, asciidoc, docbook-dtds
ArchLinux	gmp/libtommath	gmp/libtommath
		cmake, gcc/clang, asciidoc, docbook-xml
OSX	gmp/libtommath	gmp/libtommath
		cmake, gcc/llvm, asciidoc
Windows	libtommath	msvc, windows sdk, ucrt, libtommath
		cmake, curl, git, python, asciidoc, docbook-xml
		xsltproc, docbook-bundle, zip, nsis

Installation

You can download and install prebuilt packages on releases page.

You can use scripts/release.sh, it will create package for your system from source. You can also use overlay for gentoo.

Installation for Windows (Visual Studio) from source is not so trivial:

1. Install Visual Studio with components: msvc, windows sdk, ucrt.
2. Install chocolatey with components: cmake, curl, git, python, xsltproc, docbook-bundle, zip, nsis.
3. Install asciidoc.
4. Download docbook-xml-4.5.zip and unpack it to C:\ProgramData\chocolatey\lib\docbook-bundle\docbook-4.5.
5. Edit C:\ProgramData\chocolatey\lib\docbook-bundle\catalog.xml and append <nextCatalog catalog="./docbook-4.5/catalog.xml"/>.
6. Launch x64 Native Tools Command Promp as admin.

git clone "https://github.com/libtom/libtommath.git" --depth 1 && cd "libtommath"
nmake -f "makefile.msvc" install && cd ".."
set "XML_CATALOG_FILES=C:\ProgramData\chocolatey\lib\docbook-bundle\catalog.xml"
set "INCLUDE=C:\devel\include;%INCLUDE%" && set "LIB=C:\devel\lib;%LIB%"
set "PATH=C:\devel\bin;C:\Program Files\Git\bin;C:\Program Files\CMake\bin;%PATH%"
cd build && bash "..\scripts\release.sh"

Please use the following command for OSX.

export XML_CATALOG_FILES=/usr/local/etc/xml/catalog

Quick start

cd build
cmake -DLZWS_COMPRESSOR_DICTIONARY="sparse-array" ".." && cmake --build "."
echo -n "TOBEORNOTTOBEORTOBEORNOT" | ./src/cli/lzws | ./src/cli/lzws -d

Option	Values	Default	Description
LZWS_COMPRESSOR_DICTIONARY	sparse-array, linked-list	sparse-array	dictionary implementation
LZWS_BIGNUM_LIBRARY	gmp, tommath, "" (auto)	"" (auto)	bignum library
LZWS_SHARED	ON, OFF	ON	build shared binaries
LZWS_STATIC	ON, OFF	OFF	build static binaries
LZWS_CLI	ON, OFF	ON	build cli
LZWS_TESTS	ON, OFF	ON	build tests
LZWS_EXAMPLES	ON, OFF	OFF	build examples
LZWS_MAN	ON, OFF	OFF	build man
LZWS_ARGTABLE3_FALLBACK	github, github-archive	github-archive	argtable3 fallback
	bitbucket, bitbucket-archive
	gitlab, gitlab-archive
LZWS_COVERAGE	ON, OFF	OFF	enable coverage

Examples and docs

See examples, man and doc folders.

CI

Please visit scripts/test-images. See test script scripts/toolchains.sh and test script scripts/coverage_toolchains.sh for CI.

Real world testing

See lzws archive collector.

License

Copyright (c) 2016 David Bryant, 2018+ other authors, all rights reserved (see AUTHORS). Distributed under the BSD Software License (see LICENSE).

Notice

Releases with GMP provided without static linked binaries (GMP LGPL license limitation).