NATS: Fix illegal resource version from storage: 0 (#274) #275

jrovira-kumori · 2024-02-16T10:43:07Z

Fix for #274.

I am guessing this issue did not appear with K3s because the first command it runs when initializing is a CREATE /bootstrap/.... Therefore future COUNT ... calls would always return != 0.

This is not the case with Kubernetes KubeAPI which reads before writing anything and gets stuck in a loop of illegal resource version from storage: 0.

This PR fixes the issue with the NATS driver by incrementing the BucketRevision number to 1 by creating and deleting a bootstrap key if a BucketRevision of 0 is detected at startup.

brandond · 2024-02-16T15:55:39Z

Thanks for the contribution! Please sign-off your commit for DCO.

@bruth can you comment if this is the best way to do this?

jrovira-kumori · 2024-02-16T16:16:29Z

Thanks for the contribution! Please sign-off your commit for DCO.

@bruth can you comment if this is the best way to do this?

Done!

brandond · 2024-02-16T16:30:40Z

fwiw the bootstrap key is internal to k3s, I'm not sure this is the correct thing to create in kine.

nats should already create a /registry/health key on startup. Do we just need another second write to that key to increment the revision to 1?

kine/pkg/drivers/nats/backend.go

Lines 116 to 125 in 3773672

    
           // Start starts the backend. 
        
           // See https://github.com/kubernetes/kubernetes/blob/442a69c3bdf6fe8e525b05887e57d89db1e2f3a5/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go#L97 
        
           func (b *Backend) Start(ctx context.Context) error { 
        
           	if _, err := b.Create(ctx, "/registry/health", []byte(`{"health":"true"}`), 0); err != nil { 
        
           		if err != server.ErrKeyExists { 
        
           			b.l.Errorf("Failed to create health check key: %v", err) 
        
           		} 
        
           	} 
        
           	return nil 
        
           }

jrovira-kumori · 2024-02-16T16:40:51Z

The /registry/health is never added in the minimal repro or other tests I have done. It has only worked whenever I was debugging the Kine process and added a breakpoint.

There seems to be some kind of race condition going on which causes the error Failed to create health check key: context deadline exceeded to happen.

Here is an example of the Kine logs from the repro.

level=debug msg="using config &nats.Config{clientURL:\"nats://nats:4222\", clientOptions:[]nats.Option(nil), revHistory:0xa, bucket:\"kine\", replicas:1, slowThreshold:500000000, noEmbed:true, dontListen:false, serverConfig:\"\", stdoutLogging:false, host:\"nats\", port:4222, dataDir:\"\"}"
level=info msg="connecting to nats://nats:4222"
level=info msg="using bucket: kine"
level=info msg="metrics server is starting to listen at :8080"
level=info msg="starting metrics server path /metrics"
level=info msg="bucket initialized: kine"
level=error msg="Failed to create health check key: context deadline exceeded"
level=info msg="Kine available at http://127.0.0.1:2379"
level=trace msg="LIST /registry/apiextensions.k8s.io/customresourcedefinitions/, start=/registry/apiextensions.k8s.io/customresourcedefinitions/, limit=10001, rev=0 => rev=0, kvs=0, err=<nil>, duration=2.334µs"
level=trace msg="LIST key=/registry/apiextensions.k8s.io/customresourcedefinitions/, end=/registry/apiextensions.k8s.io/customresourcedefinitions/, revision=0, currentRev=0 count=0, limit=10000"
level=trace msg="COUNT /registry/apiextensions.k8s.io/customresourcedefinitions/, rev=0 => rev=0, count=0, err=<nil>, duration=461ns"
level=trace msg="LIST COUNT key=/registry/apiextensions.k8s.io/customresourcedefinitions/, end=/registry/apiextensions.k8s.io/customresourcedefinitions/, revision=0, currentRev=0 count=0"
level=trace msg="COUNT /registry/events/, rev=0 => rev=0, count=0, err=<nil>, duration=592ns"
level=trace msg="LIST COUNT key=/registry/events/, end=/registry/events/, revision=0, currentRev=0 count=0"
level=trace msg="COUNT /registry/resourcequotas/, rev=0 => rev=0, count=0, err=<nil>, duration=290ns"
level=trace msg="LIST COUNT key=/registry/resourcequotas/, end=/registry/resourcequotas/, revision=0, currentRev=0 count=0"
level=trace msg="LIST /registry/resourcequotas/, start=/registry/resourcequotas/, limit=10001, rev=0 => rev=0, kvs=0, err=<nil>, duration=7.494µs"
level=trace msg="LIST key=/registry/resourcequotas/, end=/registry/resourcequotas/, revision=0, currentRev=0 count=0, limit=10000"
level=trace msg="COUNT /registry/secrets/, rev=0 => rev=0, count=0, err=<nil>, duration=310ns"
level=trace msg="LIST COUNT key=/registry/secrets/, end=/registry/secrets/, revision=0, currentRev=0 count=0"

jrovira-kumori · 2024-02-16T16:46:58Z

fwiw the bootstrap key is internal to k3s, I'm not sure this is the correct thing to create in kine.

The constant string "bootstrap" in the commits is meaningless. I believe any string would work here as the calls are done to the nats-go KeyValue api directly.

It should also have no effect as the value is immediately deleted.

brandond · 2024-02-16T17:39:26Z

level=error msg="Failed to create health check key: context deadline exceeded"

It sounds like the root cause here is that the health check key isn't getting created. Rather than creating and deleting another key, lets just fix the bug that's causing that key to not get created?

jrovira-kumori · 2024-02-19T08:08:00Z

It sounds like the root cause here is that the health check key isn't getting created. Rather than creating and deleting another key, lets just fix the bug that's causing that key to not get created?

The context deadline exceeded error was coming from github.com/nats-io/nats.go/[email protected]. The getMsg() wraps the context in a 5 second deadline when there is no other deadline defined.

I was unable to identify why the method times out after the stream is created in NATS. Maybe someone with more in-depth knowledge on its inner workings can clarify it. Anyhow, adding some retry logic to the Start() method seems to resolve the issues that were showing up.

I have removed the previous bootstrap key creation when initializing the NATS backend in favour of the retry logic.

Signed-off-by: J. Rovira <[email protected]>

bruth · 2024-02-19T16:34:44Z

Will take a look today.

bruth · 2024-02-20T13:03:00Z

The retry does work, but I am digging into why, once the KV bucket is created, the first write seemingly times out. It may be that the bucket is actually not yet ready to receive writes, but that should not be the case so I will dig a bit deeper on it.

brandond · 2024-03-01T22:04:22Z

@bruth any update on this? I don't think we should fix this on the kine side if the issue is in nats.

bruth · 2024-03-02T12:24:50Z

The latest version of the NATS Go client appears to fix the issue. At least I can't reproduce the bug with it. I will open a PR to update the NATS deps for Kine.

jrovira-kumori · 2024-03-04T09:19:06Z

The latest version of the NATS Go client appears to fix the issue. At least I can't reproduce the bug with it. I will open a PR to update the NATS deps for Kine.

I have tried the updated version from #281. From my testing, the issue still persists.

It can be reproduced easily with just nats-server (docker) and the kine binary. On a separate terminal run NATS:

docker run --rm --network host nats:2.10.11 -js -DVV

Then compile and run the updated kine version:

git clone https://github.com/nats-io/kine
cd kine
git switch update-nats-versions
go build .
./kine --debug --endpoint "nats://?noEmbed=true"

Kine will display the same error.

ERRO[2024-03-04T10:08:56.261135233+01:00] Failed to create health check key: context deadline exceeded

Full logs:

kine

DEBU[2024-03-04T10:08:51.255193539+01:00] using config &nats.Config{clientURL:"nats://", clientOptions:[]nats.Option(nil), revHistory:0xa, bucket:"kine", replicas:1, slowThreshold:500000000, noEmbed:false, dontListen:false, serverConfig:"", stdoutLogging:false, host:"", port:0, dataDir:""} 
INFO[2024-03-04T10:08:51.255250025+01:00] connecting to nats://                        
INFO[2024-03-04T10:08:51.255258441+01:00] using bucket: kine                           
INFO[2024-03-04T10:08:51.25528472+01:00] metrics server is starting to listen at :8080 
INFO[2024-03-04T10:08:51.255458986+01:00] starting metrics server path /metrics        
INFO[2024-03-04T10:08:51.259200075+01:00] bucket initialized: kine                     
ERRO[2024-03-04T10:08:56.261135233+01:00] Failed to create health check key: context deadline exceeded 
INFO[2024-03-04T10:08:56.26128823+01:00] Kine available at http://127.0.0.1:2379      
^CERRO[2024-03-04T10:08:58.766961723+01:00] btree watcher error: context canceled

nats-server

[1] 2024/03/04 09:08:49.492112 [INF] Starting nats-server
[1] 2024/03/04 09:08:49.492208 [INF]   Version:  2.10.11
[1] 2024/03/04 09:08:49.492210 [INF]   Git:      [5fc6051]
[1] 2024/03/04 09:08:49.492213 [DBG]   Go build: go1.21.7
[1] 2024/03/04 09:08:49.492214 [INF]   Name:     NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H
[1] 2024/03/04 09:08:49.492219 [INF]   Node:     fvNdmIeF
[1] 2024/03/04 09:08:49.492221 [INF]   ID:       NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H
[1] 2024/03/04 09:08:49.492241 [DBG] Created system account: "$SYS"
[1] 2024/03/04 09:08:49.492375 [TRC] SYSTEM - <<- [SUB $SYS._INBOX.fvNdmIeF.*  2]
[1] 2024/03/04 09:08:49.492403 [TRC] SYSTEM - <<- [SUB $SYS.SERVER.ACCOUNT.*.CONNS  3]
[1] 2024/03/04 09:08:49.492409 [TRC] SYSTEM - <<- [SUB $SYS._INBOX_.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H  4]
[1] 2024/03/04 09:08:49.492418 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.NSUBS  5]
[1] 2024/03/04 09:08:49.492422 [TRC] SYSTEM - <<- [SUB $SYS.SERVER.*.STATSZ  6]
[1] 2024/03/04 09:08:49.492425 [TRC] SYSTEM - <<- [SUB $SYS.SERVER.*.SHUTDOWN  7]
[1] 2024/03/04 09:08:49.492429 [TRC] SYSTEM - <<- [SUB $SYS.SERVER.*.LAMEDUCK  8]
[1] 2024/03/04 09:08:49.492433 [TRC] SYSTEM - <<- [SUB $SYS.ACCOUNT.*.CLAIMS.UPDATE  9]
[1] 2024/03/04 09:08:49.492437 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.CLAIMS.UPDATE  10]
[1] 2024/03/04 09:08:49.492442 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING  11]
[1] 2024/03/04 09:08:49.492447 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.STATSZ  12]
[1] 2024/03/04 09:08:49.492458 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.STATSZ  13]
[1] 2024/03/04 09:08:49.492463 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.CONNZ  14]
[1] 2024/03/04 09:08:49.492470 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.CONNZ  15]
[1] 2024/03/04 09:08:49.492474 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.ROUTEZ  16]
[1] 2024/03/04 09:08:49.492485 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.ROUTEZ  17]
[1] 2024/03/04 09:08:49.492491 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.GATEWAYZ  18]
[1] 2024/03/04 09:08:49.492498 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.GATEWAYZ  19]
[1] 2024/03/04 09:08:49.492502 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.ACCOUNTZ  20]
[1] 2024/03/04 09:08:49.492505 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.ACCOUNTZ  21]
[1] 2024/03/04 09:08:49.492508 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.JSZ  22]
[1] 2024/03/04 09:08:49.492512 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.JSZ  23]
[1] 2024/03/04 09:08:49.492517 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.PROFILEZ  24]
[1] 2024/03/04 09:08:49.492521 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.PROFILEZ  25]
[1] 2024/03/04 09:08:49.492524 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.IDZ  26]
[1] 2024/03/04 09:08:49.492528 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.IDZ  27]
[1] 2024/03/04 09:08:49.492532 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.VARZ  28]
[1] 2024/03/04 09:08:49.492536 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.VARZ  29]
[1] 2024/03/04 09:08:49.492540 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.SUBSZ  30]
[1] 2024/03/04 09:08:49.492546 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.SUBSZ  31]
[1] 2024/03/04 09:08:49.492551 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.LEAFZ  32]
[1] 2024/03/04 09:08:49.492554 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.LEAFZ  33]
[1] 2024/03/04 09:08:49.492557 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.HEALTHZ  34]
[1] 2024/03/04 09:08:49.492562 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.PING.HEALTHZ  35]
[1] 2024/03/04 09:08:49.492566 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.SUBSZ  36]
[1] 2024/03/04 09:08:49.492571 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.CONNZ  37]
[1] 2024/03/04 09:08:49.492583 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.LEAFZ  38]
[1] 2024/03/04 09:08:49.492587 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.JSZ  39]
[1] 2024/03/04 09:08:49.492590 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.INFO  40]
[1] 2024/03/04 09:08:49.492594 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.STATZ  41]
[1] 2024/03/04 09:08:49.492600 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.*.CONNS  42]
[1] 2024/03/04 09:08:49.492596 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.ACCOUNT.$SYS.CONNS  535]
[1] 2024/03/04 09:08:49.492603 [TRC] SYSTEM - <<- [SUB $SYS.REQ.USER.*.INFO  43]
[1] 2024/03/04 09:08:49.492614 [TRC] SYSTEM - <<- [SUB $SYS.REQ.ACCOUNT.PING.STATZ  44]
[1] 2024/03/04 09:08:49.492619 [TRC] SYSTEM - <<- [SUB $SYS.ACCOUNT.*.LEAFNODE.CONNECT  45]
[1] 2024/03/04 09:08:49.492623 [TRC] SYSTEM - <<- [SUB $SYS.LATENCY.M2.fvNdmIeF  46]
[1] 2024/03/04 09:08:49.492607 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Q2\",\"timestamp\":\"2024-03-04T09:08:49.492278704Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":2,\"time\":\"2024-03-04T09:08:49.492405362Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":0,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"]
[1] 2024/03/04 09:08:49.492626 [TRC] SYSTEM - <<- [SUB $SYS.DEBUG.SUBSCRIBERS  47]
[1] 2024/03/04 09:08:49.492633 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.RELOAD  48]
[1] 2024/03/04 09:08:49.492638 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.KICK  49]
[1] 2024/03/04 09:08:49.492642 [TRC] SYSTEM - <<- [SUB $SYS.REQ.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.LDM  50]
[1] 2024/03/04 09:08:49.492643 [TRC] SYSTEM - <<- [PUB $SYS.ACCOUNT.$SYS.SERVER.CONNS  535]
[1] 2024/03/04 09:08:49.492649 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Q2\",\"timestamp\":\"2024-03-04T09:08:49.492278704Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":3,\"time\":\"2024-03-04T09:08:49.492634841Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":0,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"]
[1] 2024/03/04 09:08:49.492668 [INF] Starting JetStream
[1] 2024/03/04 09:08:49.492672 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.ACCOUNT.$SYS.CONNS  535]
[1] 2024/03/04 09:08:49.492675 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Rd\",\"timestamp\":\"2024-03-04T09:08:49.492392578Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":4,\"time\":\"2024-03-04T09:08:49.492664988Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":2,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"]
[1] 2024/03/04 09:08:49.492690 [TRC] SYSTEM - <<- [PUB $SYS.ACCOUNT.$SYS.SERVER.CONNS  535]
[1] 2024/03/04 09:08:49.492692 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Rd\",\"timestamp\":\"2024-03-04T09:08:49.492392578Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":5,\"time\":\"2024-03-04T09:08:49.492686588Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":2,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"]
[1] 2024/03/04 09:08:49.492721 [TRC] SYSTEM - <<- [PUB $SYS.REQ.ACCOUNT.$G.CONNS $SYS._INBOX_.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H 261]
[1] 2024/03/04 09:08:49.492724 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":6,\"time\":\"2024-03-04T09:08:49.492703761Z\"},\"acc\":\"$G\"}"]
[1] 2024/03/04 09:08:49.492737 [TRC] SYSTEM - <<- [PUB $SYS.REQ.ACCOUNT.$SYS.CONNS $SYS._INBOX_.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H 263]
[1] 2024/03/04 09:08:49.492741 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":7,\"time\":\"2024-03-04T09:08:49.492733206Z\"},\"acc\":\"$SYS\"}"]
[1] 2024/03/04 09:08:49.492842 [DBG] JetStream creating dynamic configuration - 22.99 GB memory, 468.64 GB disk
[1] 2024/03/04 09:08:49.493030 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[1] 2024/03/04 09:08:49.493036 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[1] 2024/03/04 09:08:49.493037 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[1] 2024/03/04 09:08:49.493039 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[1] 2024/03/04 09:08:49.493040 [INF] 
[1] 2024/03/04 09:08:49.493041 [INF]          https://docs.nats.io/jetstream
[1] 2024/03/04 09:08:49.493043 [INF] 
[1] 2024/03/04 09:08:49.493044 [INF] ---------------- JETSTREAM ----------------
[1] 2024/03/04 09:08:49.493047 [INF]   Max Memory:      22.99 GB
[1] 2024/03/04 09:08:49.493049 [INF]   Max Storage:     468.64 GB
[1] 2024/03/04 09:08:49.493051 [INF]   Store Directory: "/tmp/nats/jetstream"
[1] 2024/03/04 09:08:49.493053 [INF] -------------------------------------------
[1] 2024/03/04 09:08:49.493149 [TRC] SYSTEM - <<- [SUB $JS.API.>  51]
[1] 2024/03/04 09:08:49.493225 [DBG]   Exports:
[1] 2024/03/04 09:08:49.493228 [DBG]      $JS.API.>
[1] 2024/03/04 09:08:49.493236 [TRC] SYSTEM - <<- [SUB $JS.API.ACCOUNT.PURGE.*  52]
[1] 2024/03/04 09:08:49.493256 [DBG] Enabled JetStream for account "$G"
[1] 2024/03/04 09:08:49.493263 [DBG]   Max Memory:      -1 B
[1] 2024/03/04 09:08:49.493265 [DBG]   Max Storage:     -1 B
[1] 2024/03/04 09:08:49.493487 [DBG] JetStream state for account "$G" recovered
[1] 2024/03/04 09:08:49.493689 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2024/03/04 09:08:49.493693 [DBG] Get non local IPs for "0.0.0.0"
[1] 2024/03/04 09:08:49.493998 [DBG]   ip=192.168.1.113
[1] 2024/03/04 09:08:49.494055 [DBG]   ip=172.17.0.1
[1] 2024/03/04 09:08:49.494106 [DBG]   ip=192.168.48.1
[1] 2024/03/04 09:08:49.494112 [DBG]   ip=fc00:f853:ccd:e793::1
[1] 2024/03/04 09:08:49.494224 [INF] Server is ready
[1] 2024/03/04 09:08:49.494415 [DBG] maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
[1] 2024/03/04 09:08:51.255553 [DBG] 127.0.0.1:49006 - cid:5 - Client connection created
[1] 2024/03/04 09:08:51.255885 [TRC] 127.0.0.1:49006 - cid:5 - <<- [CONNECT {"verbose":false,"pedantic":false,"tls_required":false,"name":"kine using bucket: kine","lang":"go","version":"1.33.1","protocol":1,"echo":true,"headers":true,"no_responders":true}]
[1] 2024/03/04 09:08:51.255987 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PING]
[1] 2024/03/04 09:08:51.255991 [TRC] 127.0.0.1:49006 - cid:5 - ->> [PONG]
[1] 2024/03/04 09:08:51.256112 [TRC] 127.0.0.1:49006 - cid:5 - <<- [SUB _INBOX.vDzLYVH9z9NsXOYvR0UmYH.*  1]
[1] 2024/03/04 09:08:51.256126 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.INFO.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.5B0kuxeZ 0]
[1] 2024/03/04 09:08:51.256129 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""]
[1] 2024/03/04 09:08:51.256530 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.DzKUmR  158]
[1] 2024/03/04 09:08:51.256535 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_info_response\",\"error\":{\"code\":404,\"err_code\":10059,\"description\":\"stream not found\"},\"total\":0,\"offset\":0,\"limit\":0}"]
[1] 2024/03/04 09:08:51.256548 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.5B0kuxeZ 1 158]
[1] 2024/03/04 09:08:51.256553 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  692]
[1] 2024/03/04 09:08:51.256555 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWL8\",\"timestamp\":\"2024-03-04T09:08:51.256489256Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.KV_kine\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"error\\\":{\\\"code\\\":404,\\\"err_code\\\":10059,\\\"description\\\":\\\"stream not found\\\"},\\\"total\\\":0,\\\"offset\\\":0,\\\"limit\\\":0}\"}"]
[1] 2024/03/04 09:08:51.256898 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.INFO _INBOX.vDzLYVH9z9NsXOYvR0UmYH.OZsWrv8O 0]
[1] 2024/03/04 09:08:51.256904 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""]
[1] 2024/03/04 09:08:51.257048 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.UoK1hf  363]
[1] 2024/03/04 09:08:51.257052 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.account_info_response\",\"memory\":0,\"storage\":0,\"reserved_memory\":0,\"reserved_storage\":0,\"streams\":0,\"consumers\":0,\"limits\":{\"max_memory\":-1,\"max_storage\":-1,\"max_streams\":-1,\"max_consumers\":-1,\"max_ack_pending\":-1,\"memory_max_stream_bytes\":-1,\"storage_max_stream_bytes\":-1,\"max_bytes_required\":false},\"api\":{\"total\":1,\"errors\":1}}"]
[1] 2024/03/04 09:08:51.257070 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.OZsWrv8O 1 363]
[1] 2024/03/04 09:08:51.257075 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  902]
[1] 2024/03/04 09:08:51.257077 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWQL\",\"timestamp\":\"2024-03-04T09:08:51.257028406Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.INFO\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.account_info_response\\\",\\\"memory\\\":0,\\\"storage\\\":0,\\\"reserved_memory\\\":0,\\\"reserved_storage\\\":0,\\\"streams\\\":0,\\\"consumers\\\":0,\\\"limits\\\":{\\\"max_memory\\\":-1,\\\"max_storage\\\":-1,\\\"max_streams\\\":-1,\\\"max_consumers\\\":-1,\\\"max_ack_pending\\\":-1,\\\"memory_max_stream_bytes\\\":-1,\\\"storage_max_stream_bytes\\\":-1,\\\"max_bytes_required\\\":false},\\\"api\\\":{\\\"total\\\":1,\\\"errors\\\":1}}\"}"]
[1] 2024/03/04 09:08:51.257280 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.CREATE.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.HzyzdaBy 422]
[1] 2024/03/04 09:08:51.257289 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: ["{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"discard\":\"old\",\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"deny_delete\":true,\"allow_rollup_hdrs\":true,\"compression\":\"none\",\"allow_direct\":true,\"mirror_direct\":false,\"consumer_limits\":{}}"]
[1] 2024/03/04 09:08:51.257484 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.ACCOUNT.$SYS.CONNS  536]
[1] 2024/03/04 09:08:51.257492 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Up\",\"timestamp\":\"2024-03-04T09:08:51.257394773Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":8,\"time\":\"2024-03-04T09:08:51.257464002Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":53,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"]
[1] 2024/03/04 09:08:51.257518 [TRC] SYSTEM - <<- [PUB $SYS.ACCOUNT.$SYS.SERVER.CONNS  536]
[1] 2024/03/04 09:08:51.257521 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55Up\",\"timestamp\":\"2024-03-04T09:08:51.257394773Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":9,\"time\":\"2024-03-04T09:08:51.257514206Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":53,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"]
[1] 2024/03/04 09:08:51.258159 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.sSb33q  769]
[1] 2024/03/04 09:08:51.258163 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_create_response\",\"config\":{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"discard\":\"old\",\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"compression\":\"none\",\"allow_direct\":true,\"mirror_direct\":false,\"sealed\":false,\"deny_delete\":true,\"deny_purge\":false,\"allow_rollup_hdrs\":true,\"consumer_limits\":{}},\"created\":\"2024-03-04T09:08:51.257402427Z\",\"state\":{\"messages\":0,\"bytes\":0,\"first_seq\":0,\"first_ts\":\"0001-01-01T00:00:00Z\",\"last_seq\":0,\"last_ts\":\"0001-01-01T00:00:00Z\",\"consumer_count\":0},\"ts\":\"2024-03-04T09:08:51.258080497Z\",\"did_create\":true}"]
[1] 2024/03/04 09:08:51.258193 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.HzyzdaBy 1 769]
[1] 2024/03/04 09:08:51.258201 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1870]
[1] 2024/03/04 09:08:51.258205 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWal\",\"timestamp\":\"2024-03-04T09:08:51.258131753Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.CREATE.KV_kine\",\"request\":\"{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"deny_delete\\\":true,\\\"allow_rollup_hdrs\\\":true,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":true,\\\"mirror_direct\\\":false,\\\"consumer_limits\\\":{}}\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_create_response\\\",\\\"config\\\":{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":true,\\\"mirror_direct\\\":false,\\\"sealed\\\":false,\\\"deny_delete\\\":true,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":true,\\\"consumer_limits\\\":{}},\\\"created\\\":\\\"2024-03-04T09:08:51.257402427Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":0},\\\"ts\\\":\\\"2024-03-04T09:08:51.258080497Z\\\",\\\"did_create\\\":true}\"}"]
[1] 2024/03/04 09:08:51.258367 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.INFO.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.5rg88gvE 0]
[1] 2024/03/04 09:08:51.258373 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""]
[1] 2024/03/04 09:08:51.258457 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.h8FuzH  860]
[1] 2024/03/04 09:08:51.258461 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_info_response\",\"total\":0,\"offset\":0,\"limit\":0,\"config\":{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"discard\":\"old\",\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"compression\":\"none\",\"allow_direct\":true,\"mirror_direct\":false,\"sealed\":false,\"deny_delete\":true,\"deny_purge\":false,\"allow_rollup_hdrs\":true,\"consumer_limits\":{}},\"created\":\"2024-03-04T09:08:51.257402427Z\",\"state\":{\"messages\":0,\"bytes\":0,\"first_seq\":0,\"first_ts\":\"0001-01-01T00:00:00Z\",\"last_seq\":0,\"last_ts\":\"0001-01-01T00:00:00Z\",\"consumer_count\":0},\"cluster\":{\"leader\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\"},\"ts\":\"2024-03-04T09:08:51.258417549Z\"}"]
[1] 2024/03/04 09:08:51.258482 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.5rg88gvE 1 860]
[1] 2024/03/04 09:08:51.258487 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1479]
[1] 2024/03/04 09:08:51.258488 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWfy\",\"timestamp\":\"2024-03-04T09:08:51.258435783Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.KV_kine\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"total\\\":0,\\\"offset\\\":0,\\\"limit\\\":0,\\\"config\\\":{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":true,\\\"mirror_direct\\\":false,\\\"sealed\\\":false,\\\"deny_delete\\\":true,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":true,\\\"consumer_limits\\\":{}},\\\"created\\\":\\\"2024-03-04T09:08:51.257402427Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":0},\\\"cluster\\\":{\\\"leader\\\":\\\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\\\"},\\\"ts\\\":\\\"2024-03-04T09:08:51.258417549Z\\\"}\"}"]
[1] 2024/03/04 09:08:51.258613 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.UPDATE.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.UnFu5lTT 423]
[1] 2024/03/04 09:08:51.258617 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: ["{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"discard\":\"old\",\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"deny_delete\":true,\"allow_rollup_hdrs\":true,\"compression\":\"none\",\"allow_direct\":false,\"mirror_direct\":false,\"consumer_limits\":{}}"]
[1] 2024/03/04 09:08:51.258710 [TRC] JETSTREAM - <-> [DELSUB 2]
[1] 2024/03/04 09:08:51.258717 [TRC] JETSTREAM - <-> [DELSUB 3]
[1] 2024/03/04 09:08:51.259032 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.zkwcsR  751]
[1] 2024/03/04 09:08:51.259037 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_update_response\",\"config\":{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"discard\":\"old\",\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"compression\":\"none\",\"allow_direct\":false,\"mirror_direct\":false,\"sealed\":false,\"deny_delete\":true,\"deny_purge\":false,\"allow_rollup_hdrs\":true,\"consumer_limits\":{}},\"created\":\"2024-03-04T09:08:51.257402427Z\",\"state\":{\"messages\":0,\"bytes\":0,\"first_seq\":0,\"first_ts\":\"0001-01-01T00:00:00Z\",\"last_seq\":0,\"last_ts\":\"0001-01-01T00:00:00Z\",\"consumer_count\":0},\"ts\":\"2024-03-04T09:08:51.25895733Z\"}"]
[1] 2024/03/04 09:08:51.259063 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.UnFu5lTT 1 751]
[1] 2024/03/04 09:08:51.259068 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1851]
[1] 2024/03/04 09:08:51.259071 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCWqO\",\"timestamp\":\"2024-03-04T09:08:51.259004909Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.UPDATE.KV_kine\",\"request\":\"{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"deny_delete\\\":true,\\\"allow_rollup_hdrs\\\":true,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":false,\\\"mirror_direct\\\":false,\\\"consumer_limits\\\":{}}\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_update_response\\\",\\\"config\\\":{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":false,\\\"mirror_direct\\\":false,\\\"sealed\\\":false,\\\"deny_delete\\\":true,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":true,\\\"consumer_limits\\\":{}},\\\"created\\\":\\\"2024-03-04T09:08:51.257402427Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":0},\\\"ts\\\":\\\"2024-03-04T09:08:51.25895733Z\\\"}\"}"]
[1] 2024/03/04 09:08:51.259368 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.DIRECT.GET.KV_kine.$KV.kine.L8nZchjdPLL.tzCUBDXD _INBOX.vDzLYVH9z9NsXOYvR0UmYH.3Xcl954a 0]
[1] 2024/03/04 09:08:51.259376 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""]
[1] 2024/03/04 09:08:51.259528 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.CONSUMER.CREATE.KV_kine.vDzLYVH9z9NsXOYvR0UmcQ_1 _INBOX.vDzLYVH9z9NsXOYvR0UmYH.V0Q93c3O 221]
[1] 2024/03/04 09:08:51.259553 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: ["{\"stream_name\":\"KV_kine\",\"config\":{\"name\":\"vDzLYVH9z9NsXOYvR0UmcQ_1\",\"deliver_policy\":\"all\",\"ack_policy\":\"none\",\"replay_policy\":\"instant\",\"inactive_threshold\":300000000000,\"num_replicas\":1,\"mem_storage\":true},\"action\":\"\"}"]
[1] 2024/03/04 09:08:51.259860 [TRC] JETSTREAM - <<- [SUB $JSC.CI.$G.KV_kine.vDzLYVH9z9NsXOYvR0UmcQ_1  53]
[1] 2024/03/04 09:08:51.259882 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.ACCOUNT.$SYS.CONNS  534]
[1] 2024/03/04 09:08:51.259889 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55WQ\",\"timestamp\":\"2024-03-04T09:08:51.2598007Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":10,\"time\":\"2024-03-04T09:08:51.25987025Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":53,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"]
[1] 2024/03/04 09:08:51.259923 [TRC] SYSTEM - <<- [PUB $SYS.ACCOUNT.$SYS.SERVER.CONNS  535]
[1] 2024/03/04 09:08:51.259930 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.server.advisory.v1.account_connections\",\"id\":\"bF9dMPXu2LGtGvuYvL55WQ\",\"timestamp\":\"2024-03-04T09:08:51.2598007Z\",\"server\":{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":11,\"time\":\"2024-03-04T09:08:51.259913351Z\"},\"acc\":\"$SYS\",\"conns\":0,\"leafnodes\":0,\"total_conns\":0,\"num_subscriptions\":53,\"sent\":{\"msgs\":0,\"bytes\":0},\"received\":{\"msgs\":0,\"bytes\":0},\"slow_consumers\":0}"]
[1] 2024/03/04 09:08:51.260087 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.1h0csz  582]
[1] 2024/03/04 09:08:51.260092 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.consumer_create_response\",\"stream_name\":\"KV_kine\",\"name\":\"vDzLYVH9z9NsXOYvR0UmcQ_1\",\"created\":\"2024-03-04T09:08:51.259797263Z\",\"config\":{\"name\":\"vDzLYVH9z9NsXOYvR0UmcQ_1\",\"deliver_policy\":\"all\",\"ack_policy\":\"none\",\"max_deliver\":-1,\"replay_policy\":\"instant\",\"max_waiting\":512,\"inactive_threshold\":300000000000,\"num_replicas\":1,\"mem_storage\":true},\"delivered\":{\"consumer_seq\":0,\"stream_seq\":0},\"ack_floor\":{\"consumer_seq\":0,\"stream_seq\":0},\"num_ack_pending\":0,\"num_redelivered\":0,\"num_waiting\":0,\"num_pending\":0,\"ts\":\"2024-03-04T09:08:51.259876061Z\"}"]
[1] 2024/03/04 09:08:51.260112 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.V0Q93c3O 1 582]
[1] 2024/03/04 09:08:51.260119 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1459]
[1] 2024/03/04 09:08:51.260122 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCX0o\",\"timestamp\":\"2024-03-04T09:08:51.260069994Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.CONSUMER.CREATE.KV_kine.vDzLYVH9z9NsXOYvR0UmcQ_1\",\"request\":\"{\\\"stream_name\\\":\\\"KV_kine\\\",\\\"config\\\":{\\\"name\\\":\\\"vDzLYVH9z9NsXOYvR0UmcQ_1\\\",\\\"deliver_policy\\\":\\\"all\\\",\\\"ack_policy\\\":\\\"none\\\",\\\"replay_policy\\\":\\\"instant\\\",\\\"inactive_threshold\\\":300000000000,\\\"num_replicas\\\":1,\\\"mem_storage\\\":true},\\\"action\\\":\\\"\\\"}\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.consumer_create_response\\\",\\\"stream_name\\\":\\\"KV_kine\\\",\\\"name\\\":\\\"vDzLYVH9z9NsXOYvR0UmcQ_1\\\",\\\"created\\\":\\\"2024-03-04T09:08:51.259797263Z\\\",\\\"config\\\":{\\\"name\\\":\\\"vDzLYVH9z9NsXOYvR0UmcQ_1\\\",\\\"deliver_policy\\\":\\\"all\\\",\\\"ack_policy\\\":\\\"none\\\",\\\"max_deliver\\\":-1,\\\"replay_policy\\\":\\\"instant\\\",\\\"max_waiting\\\":512,\\\"inactive_threshold\\\":300000000000,\\\"num_replicas\\\":1,\\\"mem_storage\\\":true},\\\"delivered\\\":{\\\"consumer_seq\\\":0,\\\"stream_seq\\\":0},\\\"ack_floor\\\":{\\\"consumer_seq\\\":0,\\\"stream_seq\\\":0},\\\"num_ack_pending\\\":0,\\\"num_redelivered\\\":0,\\\"num_waiting\\\":0,\\\"num_pending\\\":0,\\\"ts\\\":\\\"2024-03-04T09:08:51.259876061Z\\\"}\"}"]
[1] 2024/03/04 09:08:51.260416 [TRC] 127.0.0.1:49006 - cid:5 - <<- [SUB _INBOX.vDzLYVH9z9NsXOYvR0UmgZ  2]
[1] 2024/03/04 09:08:51.260426 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.CONSUMER.MSG.NEXT.KV_kine.vDzLYVH9z9NsXOYvR0UmcQ_1 _INBOX.vDzLYVH9z9NsXOYvR0UmgZ 64]
[1] 2024/03/04 09:08:51.260430 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: ["{\"expires\":30000000000,\"batch\":500,\"idle_heartbeat\":15000000000}"]
[1] 2024/03/04 09:08:51.260461 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PUB $JS.API.STREAM.INFO.KV_kine _INBOX.vDzLYVH9z9NsXOYvR0UmYH.VU63PUKW 0]
[1] 2024/03/04 09:08:51.260466 [TRC] 127.0.0.1:49006 - cid:5 - <<- MSG_PAYLOAD: [""]
[1] 2024/03/04 09:08:51.260546 [TRC] SYSTEM - <<- [PUB _R_.uBeJS4.TdPzdJ  861]
[1] 2024/03/04 09:08:51.260550 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.api.v1.stream_info_response\",\"total\":0,\"offset\":0,\"limit\":0,\"config\":{\"name\":\"KV_kine\",\"description\":\"Holds kine key/values\",\"subjects\":[\"$KV.kine.\\u003e\"],\"retention\":\"limits\",\"max_consumers\":-1,\"max_msgs\":-1,\"max_bytes\":-1,\"max_age\":0,\"max_msgs_per_subject\":10,\"max_msg_size\":-1,\"discard\":\"old\",\"storage\":\"file\",\"num_replicas\":1,\"duplicate_window\":120000000000,\"compression\":\"none\",\"allow_direct\":false,\"mirror_direct\":false,\"sealed\":false,\"deny_delete\":true,\"deny_purge\":false,\"allow_rollup_hdrs\":true,\"consumer_limits\":{}},\"created\":\"2024-03-04T09:08:51.257402427Z\",\"state\":{\"messages\":0,\"bytes\":0,\"first_seq\":0,\"first_ts\":\"0001-01-01T00:00:00Z\",\"last_seq\":0,\"last_ts\":\"0001-01-01T00:00:00Z\",\"consumer_count\":1},\"cluster\":{\"leader\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\"},\"ts\":\"2024-03-04T09:08:51.260516591Z\"}"]
[1] 2024/03/04 09:08:51.260574 [TRC] 127.0.0.1:49006 - cid:5 - ->> [MSG _INBOX.vDzLYVH9z9NsXOYvR0UmYH.VU63PUKW 1 861]
[1] 2024/03/04 09:08:51.260581 [TRC] ACCOUNT - <<- [PUB $JS.EVENT.ADVISORY.API  1480]
[1] 2024/03/04 09:08:51.260587 [TRC] ACCOUNT - <<- MSG_PAYLOAD: ["{\"type\":\"io.nats.jetstream.advisory.v1.api_audit\",\"id\":\"KFTfI3BbLHx57EKrAUCX61\",\"timestamp\":\"2024-03-04T09:08:51.260530607Z\",\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"client\":{\"start\":\"2024-03-04T09:08:51.255511054Z\",\"host\":\"127.0.0.1\",\"id\":5,\"acc\":\"$G\",\"name\":\"kine using bucket: kine\",\"lang\":\"go\",\"ver\":\"1.33.1\",\"rtt\":407042,\"server\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"kind\":\"Client\",\"client_type\":\"nats\"},\"subject\":\"$JS.API.STREAM.INFO.KV_kine\",\"response\":\"{\\\"type\\\":\\\"io.nats.jetstream.api.v1.stream_info_response\\\",\\\"total\\\":0,\\\"offset\\\":0,\\\"limit\\\":0,\\\"config\\\":{\\\"name\\\":\\\"KV_kine\\\",\\\"description\\\":\\\"Holds kine key/values\\\",\\\"subjects\\\":[\\\"$KV.kine.\\\\u003e\\\"],\\\"retention\\\":\\\"limits\\\",\\\"max_consumers\\\":-1,\\\"max_msgs\\\":-1,\\\"max_bytes\\\":-1,\\\"max_age\\\":0,\\\"max_msgs_per_subject\\\":10,\\\"max_msg_size\\\":-1,\\\"discard\\\":\\\"old\\\",\\\"storage\\\":\\\"file\\\",\\\"num_replicas\\\":1,\\\"duplicate_window\\\":120000000000,\\\"compression\\\":\\\"none\\\",\\\"allow_direct\\\":false,\\\"mirror_direct\\\":false,\\\"sealed\\\":false,\\\"deny_delete\\\":true,\\\"deny_purge\\\":false,\\\"allow_rollup_hdrs\\\":true,\\\"consumer_limits\\\":{}},\\\"created\\\":\\\"2024-03-04T09:08:51.257402427Z\\\",\\\"state\\\":{\\\"messages\\\":0,\\\"bytes\\\":0,\\\"first_seq\\\":0,\\\"first_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"last_seq\\\":0,\\\"last_ts\\\":\\\"0001-01-01T00:00:00Z\\\",\\\"consumer_count\\\":1},\\\"cluster\\\":{\\\"leader\\\":\\\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\\\"},\\\"ts\\\":\\\"2024-03-04T09:08:51.260516591Z\\\"}\"}"]
[1] 2024/03/04 09:08:53.498681 [DBG] 127.0.0.1:49006 - cid:5 - Client Ping Timer
[1] 2024/03/04 09:08:53.498692 [TRC] 127.0.0.1:49006 - cid:5 - ->> [PING]
[1] 2024/03/04 09:08:53.498860 [TRC] 127.0.0.1:49006 - cid:5 - <<- [PONG]
[1] 2024/03/04 09:08:58.768966 [DBG] 127.0.0.1:49006 - cid:5 - Client connection closed: Client Closed
[1] 2024/03/04 09:08:58.769016 [TRC] 127.0.0.1:49006 - cid:5 - <-> [DELSUB 1]
[1] 2024/03/04 09:08:58.769026 [TRC] 127.0.0.1:49006 - cid:5 - <-> [DELSUB 2]
^C[1] 2024/03/04 09:08:59.651387 [DBG] Trapped "interrupt" signal
[1] 2024/03/04 09:08:59.701672 [TRC] SYSTEM - <<- [PUB $SYS.SERVER.NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H.SHUTDOWN  240]
[1] 2024/03/04 09:08:59.701689 [TRC] SYSTEM - <<- MSG_PAYLOAD: ["{\"name\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"host\":\"0.0.0.0\",\"id\":\"NCZDG2VQ2Z26FMKXGFAHB7ANCCP5OFR4O2WI7QCQHEZWRA43LXB76S6H\",\"ver\":\"2.10.11\",\"jetstream\":true,\"flags\":3,\"seq\":12,\"time\":\"2024-03-04T09:08:59.701632213Z\"}"]
[1] 2024/03/04 09:08:59.701728 [INF] Initiating Shutdown...
[1] 2024/03/04 09:08:59.701745 [INF] Initiating JetStream Shutdown...
[1] 2024/03/04 09:08:59.701791 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[1] 2024/03/04 09:08:59.701807 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[1] 2024/03/04 09:08:59.701819 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[1] 2024/03/04 09:08:59.701830 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[1] 2024/03/04 09:08:59.701891 [INF] JetStream Shutdown
[1] 2024/03/04 09:08:59.701899 [DBG] JETSTREAM - JetStream connection closed: Client Closed
[1] 2024/03/04 09:08:59.701957 [DBG] Client accept loop exiting..
[1] 2024/03/04 09:08:59.701946 [DBG] SYSTEM - System connection closed: Client Closed
[1] 2024/03/04 09:08:59.701982 [INF] Server Exiting..

kannanvr · 2024-03-22T07:09:54Z

Any update on this ? We are also facing the similar issue

bruth · 2024-03-22T12:31:30Z

Found the root cause and updated my previous PR: #281. @jrovira-kumori feel free to validate on your end.

jrovira-kumori · 2024-03-22T12:43:25Z

I am happy to confirm that the issue has been resolved! I have tested it again with kube-apiserver (just in case) and it runs like a charm.

Thanks for the help!

jrovira-kumori requested a review from a team as a code owner February 16, 2024 10:43

jrovira-kumori marked this pull request as draft February 16, 2024 10:48

jrovira-kumori marked this pull request as ready for review February 16, 2024 12:13

jrovira-kumori force-pushed the master branch from acd594d to d71ad2e Compare February 16, 2024 16:15

jrovira-kumori force-pushed the master branch from d71ad2e to d18e887 Compare February 19, 2024 07:55

jrovira-kumori closed this Feb 19, 2024

jrovira-kumori force-pushed the master branch from d18e887 to 3773672 Compare February 19, 2024 08:09

This comment was marked as off-topic.

Sign in to view

NATS: Fix illegal resource version from storage: 0 (k3s-io#274)

cfe5453

Signed-off-by: J. Rovira <[email protected]>

jrovira-kumori reopened this Feb 19, 2024

brandond mentioned this pull request Mar 18, 2024

Re-enable direct gets and update nats-server and nats.go versions #281

Merged

jrovira-kumori closed this Mar 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NATS: Fix illegal resource version from storage: 0 (#274) #275

NATS: Fix illegal resource version from storage: 0 (#274) #275

jrovira-kumori commented Feb 16, 2024 •

edited

Loading

brandond commented Feb 16, 2024

jrovira-kumori commented Feb 16, 2024

brandond commented Feb 16, 2024 •

edited

Loading

jrovira-kumori commented Feb 16, 2024 •

edited

Loading

jrovira-kumori commented Feb 16, 2024

brandond commented Feb 16, 2024

jrovira-kumori commented Feb 19, 2024

This comment was marked as off-topic.

bruth commented Feb 19, 2024

bruth commented Feb 20, 2024

brandond commented Mar 1, 2024

bruth commented Mar 2, 2024

jrovira-kumori commented Mar 4, 2024 •

edited

Loading

kannanvr commented Mar 22, 2024

bruth commented Mar 22, 2024

jrovira-kumori commented Mar 22, 2024

NATS: Fix illegal resource version from storage: 0 (#274) #275

NATS: Fix illegal resource version from storage: 0 (#274) #275

Conversation

jrovira-kumori commented Feb 16, 2024 • edited Loading

brandond commented Feb 16, 2024

jrovira-kumori commented Feb 16, 2024

brandond commented Feb 16, 2024 • edited Loading

jrovira-kumori commented Feb 16, 2024 • edited Loading

jrovira-kumori commented Feb 16, 2024

brandond commented Feb 16, 2024

jrovira-kumori commented Feb 19, 2024

This comment was marked as off-topic.

bruth commented Feb 19, 2024

bruth commented Feb 20, 2024

brandond commented Mar 1, 2024

bruth commented Mar 2, 2024

jrovira-kumori commented Mar 4, 2024 • edited Loading

kannanvr commented Mar 22, 2024

bruth commented Mar 22, 2024

jrovira-kumori commented Mar 22, 2024

jrovira-kumori commented Feb 16, 2024 •

edited

Loading

brandond commented Feb 16, 2024 •

edited

Loading

jrovira-kumori commented Feb 16, 2024 •

edited

Loading

jrovira-kumori commented Mar 4, 2024 •

edited

Loading