Add Streams API support to k6

[oleiade]

Problem Statement

The Streams API is a specification for JavaScript which aims to provide users with programmatically access streams of data and process them as desired by the developer. It was initially designed for networking and with the Fetch API in mind, however, in light of recent work made on both improving the handling of large files in k6 and our future HTTP API, it appears that it might be a good candidate to improve and streamline data consumption and manipulation in k6 too.

Proposal

We propose to provide a subset of the streams API as a k6 module, and expose it to both k6 internals as well as our public API. Judging by the scope of the Streams API itself, we propose that only ReadableStream would be implemented at first. If necessary at a later time, we could then implement support for writable and transformable streams too.

One of the main use case for streams as of this day would be around data processing, either from a file, or from a source in memory. It would likely allow us to consume data from various sources, without requiring to load it as a whole beforehand as it often is the case now.

Beyond the optimized resources consumption it would offer, we could also extend our existing APIs to have the ability to consume Streams. We believe beyond implementing and exposing streams to JavaScript user scripts, we would need to make sure our underlying API remains usable and intuitive from the Go side too to facilitate adoption from modules and extensions.

Problem space

• Design a File API for k6 #2977 would probably be a good first candidate to return a stream to underlying file data, as opposed to reading it as a whole.
• Proposal for a K6 stream API #592

Execution

Tasks

Beta Give feedback

1. Implement the Streams API ReadableStream #3666
   enhancement +1
   
2. Make ReadableStream (WIP) implementation pass WPT test suite #3659
   10 of 12
   +0
   
3. Expose a streams module to k6 scripts

[andrewslotin]

Closing as done.