Add Negentropy format

[fiatjaf]

https://github.com/hoytech/negentropy is a simple protocol for generic syncing of datasets between a client and a server.

While writing an implementation of it the binary format of the messages was confusing me (even though it is pretty simple) so I figured it would help others to be able to visualize it (it ended up helping me too even though I thought I already knew everything when I started writing this).

Hopefully I did everything right this time.

make doc ended up modifying some unrelated stuff.

---

One question: is there a way for fq to read input as hex from the stdin? I was testing using hex messages and kept having to use xxd to convert because I couldn't find a way to tell fq to read as hex.

[fiatjaf]

Also since my code is printing datetimes as field descriptions, the tests were succeeding here but failing on the CI because of the timezone. I've changed the testdata now to expect GMT, but I don't know of a way to make this seamless for everybody. Should I add this?

diff --git a/Makefile b/Makefile
index 81ca73f3..79a4dd00 100644
--- a/Makefile
+++ b/Makefile
@@ -15,7 +15,7 @@ test-race: always testgo-race testjq testcli
 # figure out all go packages with test files
 testgo: PKGS=$(shell find . -name "*_test.go" | xargs -n 1 dirname | sort | uniq)
 testgo: always
-	go test -timeout 20m ${RACE} ${VERBOSE} ${COVER} ${PKGS}
+	TZ=GMT go test -timeout 20m ${RACE} ${VERBOSE} ${COVER} ${PKGS}

 testgo-race: RACE=-race
 testgo-race: testgo

Or should I stop printing datetimes?

[wader]

https://github.com/hoytech/negentropy is a simple protocol for generic syncing of datasets between a client and a server.

While writing an implementation of it the binary format of the messages was confusing me (even though it is pretty simple) so I figured it would help others to be able to visualize it (it ended up helping me too even though I thought I already knew everything when I started writing this).

Sounds like exactly like my use case for fq and glad to hear it was useful!

Hopefully I did everything right this time.

Had a quick look and it looks good, i will have closer look soon

make doc ended up modifying some unrelated stuff.

Huh was sure it was lastest was part of the midi PR, can be part of this PR, no worries

One question: is there a way for fq to read input as hex from the stdin? I was testing using hex messages and kept having to use xxd to convert because I couldn't find a way to tell fq to read as hex.

Yes there is. fq has full jq interpreter with some extra functions and also each decoder is available as a function. So you can do things like this:

$ xxd -ps ints.msgpack | fq -o line_bytes=16 -Rs 'split("\n") | add | from_hex | msgpack'
    │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│.{}: (msgpack)
0x00│dc                                             │.               │  type: "array16" (0xdc)
0x00│   00 1a                                       │ ..             │  length: 26
0x00│         00 01 02 03 04 05 06 07 08 09 7f cc 80│   .............│  elements[0:26]:
0x10│ff fe fd fc fb fa f9 f8 e1 e0 ce ff ff ff ff d3│................│
*   │until 0x31.7 (end) (47)                        │                │

See https://github.com/wader/fq/blob/master/doc/usage.md#encodings-serializations-and-hashes for more details and decoding/encoding functions.

[wader]

Also since my code is printing datetimes as field descriptions, the tests were succeeding here but failing on the CI because of the timezone. I've changed the testdata now to expect GMT, but I don't know of a way to make this seamless for everybody. Should I add this?

diff --git a/Makefile b/Makefile
index 81ca73f3..79a4dd00 100644
--- a/Makefile
+++ b/Makefile
@@ -15,7 +15,7 @@ test-race: always testgo-race testjq testcli
 # figure out all go packages with test files
 testgo: PKGS=$(shell find . -name "*_test.go" | xargs -n 1 dirname | sort | uniq)
 testgo: always
-	go test -timeout 20m ${RACE} ${VERBOSE} ${COVER} ${PKGS}
+	TZ=GMT go test -timeout 20m ${RACE} ${VERBOSE} ${COVER} ${PKGS}

 testgo-race: RACE=-race
 testgo-race: testgo

Or should I stop printing datetimes?

Hmm it would be great if the default output was not dependent on environment and will probably be a mess for other package maintainers if not. There has been similar issue with other formats and then i think the solution was to output timestamp description in UTC, would that work? and if you really want local time etc you could add a format option to set it, or alternatively give some examples how to use the jq date/time functions to produce a local timestamp

[fiatjaf]

Yes there is. fq has full jq interpreter with some extra functions and also each decoder is available as a function. So you can do things like this:

$ xxd -ps ints.msgpack | fq -o line_bytes=16 -Rs 'split("\n") | add | from_hex | msgpack'
│00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│.{}: (msgpack)
0x00│dc │. │ type: "array16" (0xdc)
0x00│ 00 1a │ .. │ length: 26
0x00│ 00 01 02 03 04 05 06 07 08 09 7f cc 80│ .............│ elements[0:26]:
0x10│ff fe fd fc fb fa f9 f8 e1 e0 ce ff ff ff ff d3│................│

• │until 0x31.7 (end) (47) │ │
  See master/doc/usage.md#encodings-serializations-and-hashes for more details and decoding/encoding functions.

Oh, cool! So this is what I wanted for my tests (somewhat simpler):

echo '6186b7abb47c0001108e4206828ee3bf34258465809a337c6c00019a68e37b177a50b3ae7164ccc628b962020114019c1381281c9e3849d5fbd514b7bb65ad0101e601fbf7451f5d22e7fa36ae3e910e9f5215020157014a1b26853e06e9c32eb41b1df4f9ab300201e6011840e273c84bb1344f1d4e15d9aa67920200016f12ee2340888653f10b0ec2d438ac9f0101840156d2d796f4dff004ab369b9bcfa4d81e020187013f1b3c8a019800d5764e2de6bdfd2785020114017caaf0acb5dfe249aa0f7f742402168a01018301e7b8c4decb1eae455ca5714281e3245302017a01409c22636b097362df125ddffb6d944302015b01f332208bee82acf8ed922853ee54057f020001fc3e51fdb0b92966e38017f7959903850101cc01428ce0c96d49f15b50143e4fb228cb9300000131712d30e5296a7a45d07bba452d61cd' | fq -R 'from_hex | negentropy | dd'

I had skimmed that usage document and seen the from_hex function, but it was useless to me as I didn't know about the -R trick nor that the decoders were available as functions. Thank you.

[fiatjaf]

Hmm it would be great if the default output was not dependent on environment and will probably be a mess for other package maintainers if not. There has been similar issue with other formats and then i think the solution was to output timestamp description in UTC, would that work? and if you really want local time etc you could add a format option to set it, or alternatively give some examples how to use the jq date/time functions to produce a local timestamp

I've changed it to display the timestamp description to be always UTC. In the meantime I learned about the difference between symbolic and actual values and fixed one other thing. I think it's much better now.

Later I'll explore the decoder-specific options more (another thing I didn't know), but I think it's not necessary for now.

[wader]

Oh, cool! So this is what I wanted for my tests (somewhat simpler):

echo '6186b7abb47c0001108e4206828ee3bf34258465809a337c6c00019a68e37b177a50b3ae7164ccc628b962020114019c1381281c9e3849d5fbd514b7bb65ad0101e601fbf7451f5d22e7fa36ae3e910e9f5215020157014a1b26853e06e9c32eb41b1df4f9ab300201e6011840e273c84bb1344f1d4e15d9aa67920200016f12ee2340888653f10b0ec2d438ac9f0101840156d2d796f4dff004ab369b9bcfa4d81e020187013f1b3c8a019800d5764e2de6bdfd2785020114017caaf0acb5dfe249aa0f7f742402168a01018301e7b8c4decb1eae455ca5714281e3245302017a01409c22636b097362df125ddffb6d944302015b01f332208bee82acf8ed922853ee54057f020001fc3e51fdb0b92966e38017f7959903850101cc01428ce0c96d49f15b50143e4fb228cb9300000131712d30e5296a7a45d07bba452d61cd' | fq -R 'from_hex | negentropy | dd'

👍 ah yes no need for -s if things are on one line, -Rs slurps all input into input one string. Maybe also good to know that all these are copied from how jq works.

I had skimmed that usage document and seen the from_hex function, but it was useless to me as I didn't know about the -R trick nor that the decoders were available as functions. Thank you.

No worries! yeah those things are not easily discoverable atm :( maybe should work on some guide or tutorial how to work with fq

[wader]

Hmm it would be great if the default output was not dependent on environment and will probably be a mess for other package maintainers if not. There has been similar issue with other formats and then i think the solution was to output timestamp description in UTC, would that work? and if you really want local time etc you could add a format option to set it, or alternatively give some examples how to use the jq date/time functions to produce a local timestamp

I've changed it to display the timestamp description to be always UTC. In the meantime I learned about the difference between symbolic and actual values and fixed one other thing. I think it's much better now.

Yeap! added a comment about different ways but i think the current way works fine unless you want to change it.

Later I'll explore the decoder-specific options more (another thing I didn't know), but I think it's not necessary for now.

👍

[wader]

Look good to me! let me know if your ready

[fiatjaf]

I'm ready.

[wader]

Thanks!

[fiatjaf]

https://github.com/hoytech/negentropy#protocol-debugging-with-fq

[wader]

🥳 what are you building using RBSR? have to read up a bit about it, seems useful!

[fiatjaf]

I'm adding it to my Nostr library (as you can see from a giant stream of commits fixing stupid implementation bugs) and relay framework.