plans to add streaming API?

[charles-cooper]

Are there any plans to add support for the new LZ4 Streaming API? This would be useful for programs which don't want to strictly decompress an entire stream in memory.

http://fastcompression.blogspot.com/2014/05/streaming-api-for-lz4.html

[jjl]

I've started integrating this in my 'frame' branch jjl@65c9823

[mwotton]

That's great to hear, please PR when you're done :)

On Sun, 10 Apr 2016 12:19 pm James Laver [email protected] wrote:

I've started integrating this in my 'frame' branch jjl/lz4hs@65c9823
jjl@65c9823

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#12 (comment)

[charles-cooper]

@jjl you may be interested in some work I did in https://github.com/charles-cooper/lz4hs/blob/master/Codec/Compression/LZ4/Decompress.hs

[jjl]

[@charles-cooper thanks for that. My haskell FFI knowledge is a bit limited, so I'm looking forward to cribbing from your attempt :)

Just to manage expectations, I'm anticipating closing this off sometime in April depending on how my other work goes.

Some of my notes so far:

• Opaque C types are annoying. Did I do it right?
• For backcompat, I haven't removed the existing code
• There doesn't seem to be a way to apply HC over streaming

My use case is a special snowflake. I'm aiming to have 64kb max blocks whereby I'd like to aim for 64kb compressed blocks, but rather than overflow the remaining data, i'd like to detect that the last compressed piece would overflow the buffer and keep it behind for the next block so that data isn't split. I'm intending to use it in conjunction with a Rope of ByteStrings.

What are other people looking to use this for? I'd like to bear this in mind when I finish the work.

Cheers

[nh2]

Note I have picked up this task of writing an LZ4 frame implementation as part of a conduit (looking into what you guys have in your branches of course). Not quite done yet (compression works and is compatible with the lz4 command line utility, but I still have to do decompression), but I thought I'd leave that here in order to avoid duplicate work.