Piping with lrzip next

Pipes and lrzip-next

lrzip-next is able to receive input from a pipe as well as output to a file, stdout, or another program. An interesting artifact of this is memory usage.

• When lrzip-next compresses a file AND writes to a file, it will have the largest compression window
• When lrzip-next reads info from STDIN and writes to a file, its compression window will be half as large
• When lrzip-next reads from STDIN and writes to STDOUT, its compression window will again be halved

Generally, lrzip-next will reserve 2/3 of total ram for a compression window. When STDIN is used it will reserve 1/3 of ram for the compression window. When both STDIN and STDOUT are used, it will reserve 1/6 of total ram for a the compression window.

Compression

So, when a command like this is used,

tar -I "lrzip-next [*lrzip options*]" -cf *file.tar.lrz* [ *tar options* ] tar files/directories

lrzip-next will reserve 1/6 of total ram for a compression window since both STDIN and STDOUT are used. On a 16GB system, lrzip-next reports
Heuristically Computed Compression Window: 26 = 2600MB

If, on the other hand, we pipe tar to lrzip-next, a larger window is reserved, as in this command using a pipe

tar -cf- file/directories | lrzip-next [lrzip options] -o lrzip-file.lrz Note, the -o file.lrz option is critical.
Heuristically Computed Compression Window: 52 = 5200MB

And filnally with no STDIN or STDOUT (i.e. standard usage), lrzip-next will use the most ram for its compression window.
Heuristically Computed Compression Window: 105 = 10500MB

Decompression

Decompression is not as impacted by memory constraints. But it is performed in the same way.

lrzip-next -d -o- | tar -xf- files/directories where -o- is output to STDOUT by lrzip-next and -xf- is read from STDIN by tar.

Both tar and lrzip-next can use verbosity options. With tar, a listing of contents can be achieved with the -t option as in

lrzip-next -d -o- | tar -tf- files/directories where -o- is output to STDOUT by lrzip-next and -tf- is read from STDIN by tar.

Or, the usage with ``tar -I "lrzip-next" -xf file.tar.lrz` as described in Using tar with lrzip-next.