feat(cbor/unstable): introduce `@std/cbor` #5909

BlackAsLight · 2024-09-04T10:40:07Z

This pull request introduces a CBOR implementation based off the RFC 8949 spec from scratch. It introduces functions like encodeCbor, encodeCborSequence, decodeCbor, and decodeCborSequence, and a CborTag class to provide additional semantic information.

It also introduces streaming versions called:

CborSequenceEncoderStream
CborByteEncoderStream
CborTextEncoderStream
CborArrayEncoderStream
CborMapEncoderStream
CborSequenceDecoderStream
CborByteDecodedStream
CborTextDecodedStream
CborArrayDecodedStream
CborMapDecodedStream

It should be noted the different naming convention used between the "encoder", "decoder" and "decoded". The "encoder" and "decoder" classes are TransformStreams, while the "decoded" classes are ReadableStreams and act merely as a way for the user to figure out what type they have.

Due to the way streams work, if one of the "decoded" streams are yielded then they'll need to either be entirely consumed or cancelled before the next value will be yielded. This should be noted when using things like Array.fromAsync. Such a function would work only if you can guarantee that no value yielded will be one of these "decoded" streams. If such a value is yield in such a function then it will hang indefinitely.

Example 1

import { assert, assertEquals } from "@std/assert";
import { decodeCbor, encodeCbor } from "@std/cbor";

const rawMessage = [
  "Hello World",
  35,
  0.5,
  false,
  -1,
  null,
  Uint8Array.from([0, 1, 2, 3]),
];

const encodedMessage = encodeCbor(rawMessage);
const decodedMessage = decodeCbor(encodedMessage);

assert(decodedMessage instanceof Array);
assertEquals(decodedMessage, rawMessage);

Example 2

import { assert, assertEquals } from "@std/assert";
import { CborTextEncoderStream, CborTextDecodedStream, CborSequenceDecoderStream } from "@std/cbor";

const reader = CborTextEncoderStream.from(["Hello World!"])
  .readable
  .pipeThrough(new CborSequenceDecoderStream()).getReader();

const { done, value } = await reader.read();
assert(done === false);
assert(value instanceof CborTextDecodedStream);
assertEquals((await Array.fromAsync(value)).join(''), "Hello World!")

assert((await reader.read()).done === true);
reader.releaseLock();

Example 3

import { encodeBase64Url } from "@std/encoding";
import {
  CborArrayDecodedStream,
  CborArrayEncoderStream,
  CborByteDecodedStream,
  CborByteEncoderStream,
  CborMapDecodedStream,
  CborMapEncoderStream,
  type CborOutputStream,
  CborSequenceDecoderStream,
  CborSequenceEncoderStream,
  CborTag,
  CborTextDecodedStream,
  CborTextEncoderStream,
} from "@std/cbor";

const rawMessage = [
  undefined,
  null,
  true,
  false,
  3.14,
  5,
  2n ** 32n,
  "Hello World",
  new Uint8Array(25),
  new Date(),
  new CborTag(33, encodeBase64Url(new Uint8Array(7))),
  ["cake", "carrot"],
  { a: 3, b: "d" },
  CborByteEncoderStream.from([new Uint8Array(7)]),
  CborTextEncoderStream.from(["Bye!"]),
  CborArrayEncoderStream.from([
    "Hey!",
    CborByteEncoderStream.from([new Uint8Array(18)]),
  ]),
  CborMapEncoderStream.from([
    ["a", 0],
    ["b", "potato"],
  ]),
];

async function logValue(value: CborOutputStream) {
  if (
    value instanceof CborByteDecodedStream ||
    value instanceof CborTextDecodedStream
  ) {
    for await (const x of value) console.log(x);
  } else if (value instanceof CborArrayDecodedStream) {
    for await (const x of value) logValue(x);
  } else if (value instanceof CborMapDecodedStream) {
    for await (const [k, v] of value) {
      console.log(k);
      logValue(v);
    }
  } else if (value instanceof CborTag) {
    console.log(value);
    logValue(value.tagContent);
  } else console.log(value);
}

for await (
  const value of ReadableStream.from(rawMessage)
    .pipeThrough(new CborSequenceEncoderStream())
    .pipeThrough(new CborSequenceDecoderStream())
) {
  logValue(value);
}

codecov · 2024-09-04T10:44:56Z

Codecov Report

Attention: Patch coverage is 90.26275% with 126 lines in your changes missing coverage. Please review.

Project coverage is 96.55%. Comparing base (065296c) to head (ec68c5c).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
cbor/sequence_decoder_stream.ts	75.78%	83 Missing and 2 partials ⚠️
cbor/sequence_encoder_stream.ts	85.82%	18 Missing ⚠️
cbor/_common_decode.ts	95.81%	10 Missing ⚠️
cbor/_common.ts	90.66%	7 Missing ⚠️
cbor/_common_encode.ts	96.63%	4 Missing ⚠️
cbor/byte_encoder_stream.ts	97.95%	1 Missing ⚠️
cbor/text_encoder_stream.ts	98.03%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5909      +/-   ##
==========================================
- Coverage   96.75%   96.55%   -0.21%     
==========================================
  Files         509      530      +21     
  Lines       39175    40483    +1308     
  Branches     5795     6067     +272     
==========================================
+ Hits        37905    39087    +1182     
- Misses       1228     1352     +124     
- Partials       42       44       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

kt3k · 2024-09-05T05:35:59Z

I wonder if these need to be classes when they both don't have any instance states.

How about making them just deocde and encode functions? That mirrors the structure of std/msgpack

BlackAsLight · 2024-09-05T06:21:50Z

I wonder if these need to be classes when they both don't have any instance states.

How about making them just deocde and encode functions? That mirrors the structure of std/msgpack

Sure. I was trying to mirror the structure of TextEncoder/TextDecoder, but this other structure seems more common in the std.

kt3k · 2024-09-05T06:35:15Z

Side note: TextDecoder stores some chunk from the previous decode when stream option is specified. Probably that is why it's implemented as a class

…strings

iuioiua · 2024-09-09T03:24:01Z

@BlackAsLight, could you please draft this PR and un-draft once ready for us to review?

BlackAsLight · 2024-09-23T01:01:25Z

Is there a way I can get the Sentry bot to update or create a new report on what lines are missing?

cbor/deno.json

kt3k · 2024-09-24T02:46:31Z

@BlackAsLight

Coverage diff of this PR should be visible in this page https://app.codecov.io/gh/denoland/std/pull/5909

CborSequenceDecoderStream
CborByteDecodedStream
CborTextDecodedStream
CborArrayDecodedStream
CborMapDecodedStream

There are mixed naming styles, FooDecoderStream and FooDecodedStream. Are these differences intentional?

We'd like to export a single API from a single file. Can you split files based on exported APIs

encode_cbor.ts exports encodeCbor
encode_cbor_sequence.ts exports encodeCborSequence
tag.ts exports CborTag
sequence_encoder.ts exports CborSequenceEncoderStream
byte_encoder_stream.ts exports CborByteEncoderStream
text_encoder_stream.ts exports CborTextEncoderStream
array_encoder_stream.ts exports CborArrayEncoderStream
map_encoder_stream.ts exports CborMapEncoderStream
decode_cbor.ts exports decodeCbor
decode_cbor_sequence.ts exports decodeCborSequence
sequence_decoder_stream.ts exports CborSequenceDecoderStream
byte_decoded_stream.ts exports CborByteDecodedStream
text_decoded_stream exports CborTextDecodedStream
array_decoded_stream exports CborArrayDecodedStream
map_decoded_stream exports CborMapDecodedStream

BlackAsLight · 2024-09-24T03:00:27Z

@BlackAsLight

Coverage diff of this PR should be visible in this page https://app.codecov.io/gh/denoland/std/pull/5909

That page hasn't gotten updated in like 5 days now, which is why I asked.

CborSequenceDecoderStream
CborByteDecodedStream
CborTextDecodedStream
CborArrayDecodedStream
CborMapDecodedStream

There are mixed naming styles, FooDecoderStream and FooDecodedStream. Are these differences intentional?

The difference in naming style is intentional. The "Encoder" and "Decoder" ones are TransformStreams that actually do the work of converting, while the "Decoded" ones are ReadableStreams that act as a simple wrapper so the user of the lib is able to know what type of chunks to expect.

I did explore the idea of having the "Decoded" ones also be TransformStreams and handling the logic of conversion, but the logic there seemed too complicated as you don't know how far to go until you've actually decoded it.

I also explored the idea of having the other "Encoder" ones (i.e. CborByteEncodedStream) act as a mere wrapper for CborSequenceEncoderStream, but that then seemed redundant if all you wanted to send for example a byte string, as more checks would essentially need to be done for no reason.

We'd like to export a single API from a single file. Can you split files based on exported APIs

* `encode_cbor.ts` exports `encodeCbor`

* `encode_cbor_sequence.ts` exports `encodeCborSequence`

* `tag.ts` exports `CborTag`

* `sequence_encoder.ts` exports `CborSequenceEncoderStream`

* `byte_encoder_stream.ts` exports `CborByteEncoderStream`

* `text_encoder_stream.ts` exports `CborTextEncoderStream`

* `array_encoder_stream.ts` exports `CborArrayEncoderStream`

* `map_encoder_stream.ts` exports `CborMapEncoderStream`

* `decode_cbor.ts` exports `decodeCbor`

* `decode_cbor_sequence.ts` exports `decodeCborSequence`

* `sequence_decoder_stream.ts` exports `CborSequenceDecoderStream`

* `byte_decoded_stream.ts` exports `CborByteDecodedStream`

* `text_decoded_stream` exports `CborTextDecodedStream`

* `array_decoded_stream` exports `CborArrayDecodedStream`

* `map_decoded_stream` exports `CborMapDecodedStream`

That is a lot of files, but I can do that.

…er resolve

kt3k · 2024-10-08T10:55:03Z

cbor/types.ts

+ * Specifies the encodable value types for the {@link CborSequenceEncoderStream}
+ * and {@link CborArrayEncoderStream}.
+ */
+export type CborInputStream =


How about renaming this to CborStreamInput as this itself is not always a stream, but an input for the stream?

The same suggestion could also apply to:

CborMapInputStream -> CborMapStreamInput

CborOutputStream -> CborStreamOutput

CborMapOutputStream -> CborMapStreamOutput

I'll also work on this change

kt3k · 2024-10-08T11:12:09Z

I'll move the files *_decoded_stream.ts to the private paths (_*_decoded_stream.ts) and remove them from exports entries in cbor/deno.json as they are not intended to be used directly from the users. Let me know if you disagree with this change.

BlackAsLight · 2024-10-08T12:46:32Z

I'll move the files *_decoded_stream.ts to the private paths (_*_decoded_stream.ts) and remove them from exports entries in cbor/deno.json as they are not intended to be used directly from the users. Let me know if you disagree with this change.

The user isn't meant to create instances of the decoded streams themselves, but they do need access to them to be able to figure out what type they have. For example with an entry instanceof ByteDecodedStream.

Edit: maybe a Boolean function instead could be used instead by the user? isByteDecodedStream(entry). That's way there is no risk of incorrect use of the classes?

kt3k · 2024-10-08T13:14:52Z

The user isn't meant to create instances of the decoded streams themselves, but they do need access to them to be able to figure out what type they have. For example with an entry instanceof ByteDecodedStream.

instanceof check makes sense. Currently they are still exported from mod.ts. Maybe it's also good to export them from relevant decoder streams' endpoints.

kt3k · 2024-10-08T13:45:46Z

@BlackAsLight
I added the exports of CborArrayDecodedStream, CborByteDecodedStream, CborMapDecodedStream, CborTextDecodedStream from cbor/sequence-decoder-stream. What do you think?

BlackAsLight · 2024-10-08T18:25:21Z

@BlackAsLight

I added the exports of CborArrayDecodedStream, CborByteDecodedStream, CborMapDecodedStream, CborTextDecodedStream from cbor/sequence-decoder-stream. What do you think?

Looks good to me

kt3k

LGTM

BlackAsLight added 9 commits September 4, 2024 19:39

setup(cbor): workspace to look for cbor/

4f07166

feat(cbor): new CborEncoder()

a410e42

feat(cbor): new CborDecoder()

953a60d

fix(cbor): @module being in the wrong file

4c295c2

chore(cbor): deno fmt

5e96fc0

docs(cbor): Updated CborTag docs

b36ce13

fix(cbor): link:docs

3a330ba

docs(cbor): Updated CborEncoder docs

0a6eb45

docs(cbor): Updated CborDecoder docs

b5d5f3c

BlackAsLight requested a review from kt3k as a code owner September 4, 2024 10:40

Merge branch 'main' into cbor

794a4a7

BlackAsLight added 2 commits September 4, 2024 20:56

github(cbor): Update GitHub Action to recognise CBOR

b544f7f

github(cbor): Update GitHub Action to recognise cbor/

6c59514

kt3k mentioned this pull request Sep 5, 2024

chore: add settings for new @std/cbor package #5915

Merged

Merge branch 'main' into cbor

5228297

github-actions bot added the cbor label Sep 5, 2024

BlackAsLight added 7 commits September 8, 2024 20:53

refactor(cbor): CborEncoder to encodeCbor

8c597e7

feat(cbor): CborEncoderStreams

29ef8ac

fix(cbor): random bug _maybe_

b479d7d

fix(cbor): bug where CborTextEncoderStream was filtering out empty …

2a746b9

…strings

chore(cbor): fmt

ace4b9c

Merge branch 'main' into cbor

35b5b1e

refactor(cbor): CborDecoder to decodeCbor

b851450

BlackAsLight closed this Sep 9, 2024

BlackAsLight reopened this Sep 9, 2024

BlackAsLight marked this pull request as ready for review September 22, 2024 01:57

kt3k reviewed Sep 23, 2024

View reviewed changes

cbor/deno.json Outdated Show resolved Hide resolved

cbor/deno.json Outdated Show resolved Hide resolved

BlackAsLight added 3 commits September 24, 2024 06:10

fix(cbor): deno.json missing export & bumped minor version

ec405c5

fix(cbor): import_map

89e9b53

Merge branch 'main' into cbor

1a0f200

BlackAsLight added 8 commits September 24, 2024 13:56

refactor(cbor): so every API has it's own file.

efe8438

fix(cbor): bug where new Uint8Array(0) would cause a promise to nev…

a9987c4

…er resolve

tests(cbor): improved to make them more uniform

1cb2a8b

refactor(cbor): upgradeStreamFromGen()

4ec394e

Merge branch 'main' into cbor

8b3afc6

replace(cbor): upgradeStreamFromGen with toByteStream

d2acc77

Merge branch 'main' into cbor

ff7bb4e

fix(cbor): bug where CborTextEncoderStream filtered out empty strings

2502a75

kt3k reviewed Oct 8, 2024

View reviewed changes

make decoded streams non-entrypoint

d7ff661

kt3k added 3 commits October 8, 2024 14:22

rename Cbor(Map){Output,Input}Stream to Cbor(Map)Stream{Output,Input}

de9e89f

reduce max string length in testing

edc7f82

add export of decoded streams from cbor/sequence-decoder-stream

ec68c5c

kt3k approved these changes Oct 9, 2024

View reviewed changes

kt3k changed the title ~~feat(cbor/unstable): Introduce @std/cbor~~ feat(cbor/unstable): introduce @std/cbor Oct 9, 2024

kt3k merged commit 0f9a23c into denoland:main Oct 9, 2024
17 checks passed

BlackAsLight deleted the cbor branch October 11, 2024 07:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cbor/unstable): introduce `@std/cbor` #5909

feat(cbor/unstable): introduce `@std/cbor` #5909

BlackAsLight commented Sep 4, 2024 •

edited

Loading

codecov bot commented Sep 4, 2024 •

edited

Loading

kt3k commented Sep 5, 2024

BlackAsLight commented Sep 5, 2024

kt3k commented Sep 5, 2024

iuioiua commented Sep 9, 2024 •

edited

Loading

BlackAsLight commented Sep 23, 2024

kt3k commented Sep 24, 2024

BlackAsLight commented Sep 24, 2024

kt3k Oct 8, 2024

kt3k Oct 8, 2024

kt3k commented Oct 8, 2024

BlackAsLight commented Oct 8, 2024 •

edited

Loading

kt3k commented Oct 8, 2024

kt3k commented Oct 8, 2024

BlackAsLight commented Oct 8, 2024

kt3k left a comment

feat(cbor/unstable): introduce @std/cbor #5909

feat(cbor/unstable): introduce @std/cbor #5909

Conversation

BlackAsLight commented Sep 4, 2024 • edited Loading

Example 1

Example 2

Example 3

codecov bot commented Sep 4, 2024 • edited Loading

Codecov Report

kt3k commented Sep 5, 2024

BlackAsLight commented Sep 5, 2024

kt3k commented Sep 5, 2024

iuioiua commented Sep 9, 2024 • edited Loading

BlackAsLight commented Sep 23, 2024

kt3k commented Sep 24, 2024

BlackAsLight commented Sep 24, 2024

kt3k Oct 8, 2024

Choose a reason for hiding this comment

kt3k Oct 8, 2024

Choose a reason for hiding this comment

kt3k commented Oct 8, 2024

BlackAsLight commented Oct 8, 2024 • edited Loading

kt3k commented Oct 8, 2024

kt3k commented Oct 8, 2024

BlackAsLight commented Oct 8, 2024

kt3k left a comment

Choose a reason for hiding this comment

feat(cbor/unstable): introduce `@std/cbor` #5909

feat(cbor/unstable): introduce `@std/cbor` #5909

BlackAsLight commented Sep 4, 2024 •

edited

Loading

codecov bot commented Sep 4, 2024 •

edited

Loading

iuioiua commented Sep 9, 2024 •

edited

Loading

BlackAsLight commented Oct 8, 2024 •

edited

Loading