Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow controlling the setting at which libdeflate only emits an uncompressed deflate stream #67

Open
astei opened this issue Jun 29, 2020 · 2 comments

Comments

@astei
Copy link
Contributor

astei commented Jun 29, 2020

libdeflate has truly entered the mainstream when it pops up in Minecraft, as I do in Velocity 1.1.0. In Minecraft, there is a need for fast, real-time compression, and libdeflate fills an important need for accelerating the zlib/deflate-based compression used by the game's protocol.

I recently ran across a need to use libdeflate in the protocol of Minecraft: Bedrock Edition. This game is... strange, and it's not a natural fit for libdeflate's block-based decompression. In this particular edition of the game, though, everything is deflate-compressed and this wastes CPU. While libdeflate by default only emits uncompressed deflate streams for inputs that are less than 16 bytes in length, this still wastes quite a bit of CPU on our end for the small packets the game often sends and we had a need to raise this to 128 bytes. For now, we run a modified libdeflate where we patched the relevant check but we'd like to make it configurable if possible so we can run vanilla libdeflate.

Thanks for making this library, it's greatly appreciated.

@ebiggers
Copy link
Owner

ebiggers commented Jul 3, 2020

Which compression level are you using? I think the cutoff to skip compression shouldn't be fixed at 16 bytes, but rather it should depend on the compression level. The lower (faster) the level, the higher the cutoff should be. That would be an easy change to make.

But, presumably you're interested in minimizing decompression time too, not just compression time. So that would be an argument for also making this configurable independent of compression level.

There's also been a request to allow user control over the minimum match length (#57). Probably the way to go to support both of these requests would be to add a new function to allocate a libdeflate_compressor which allows for future extension, e.g.:

    struct libdeflate_compression_params {
            uint32_t level;
            uint32_t min_match_len;
            uint32_t min_compressed_block_len;
    };

    LIBDEFLATEEXPORT struct libdeflate_compressor * LIBDEFLATEAPI
    libdeflate_alloc_compressor_ex(const struct libdeflate_compression_params *params,
                                   size_t params_size);

@astei
Copy link
Contributor Author

astei commented Jul 3, 2020

We currently use compression level 4 (planning to raise this to 6). Compression time and decompression time are important, since a lot of people play on lower-end devices (though we don't control the clients at all).

A libdeflate_alloc_compressor_ex function sounds like something that would be a good idea.

ebiggers added a commit that referenced this issue Oct 18, 2020
The cutoff for outputting uncompressed data is currently < 16 bytes for
all compression levels.  That isn't ideal, since the higher the
compression level, the more we should bother with very small inputs; and
the lower the compression level, the less we should bother.

Use a formula that produces the following cutoffs:

        Level  Cutoff
        -----  ------
        0      56
        1      52
        2      48
        3      44
        4      40
        5      36
        6      32
        7      28
        8      24
        9      20
        10     16
        11     12
        12     8

Update #67
ebiggers added a commit that referenced this issue Oct 19, 2020
The cutoff for outputting uncompressed data is currently < 16 bytes for
all compression levels.  That isn't ideal, since the higher the
compression level, the more we should bother with very small inputs; and
the lower the compression level, the less we should bother.

Use a formula that produces the following cutoffs:

        Level  Cutoff
        -----  ------
        0      56
        1      52
        2      48
        3      44
        4      40
        5      36
        6      32
        7      28
        8      24
        9      20
        10     16
        11     12
        12     8

Update #67
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants