NeoFS block storage: add uploading commands #3582

AliceInHunterland · 2024-09-11T06:13:07Z

example usage:

./bin/neo-go util upload-bin --cid 9iVfUg8aDHKjPC4LhQXEkVUM4HDkR7UCXYLs8NQwYfSG --wallet-config ./wallet-config.yml --block-attribute "test_final2345678901234" --index-attribute "oid-index" --rpc-endpoint https://rpc.t5.n3.nspcc.ru:20331 -fsr st1.t5.fs.neo.org:8080

./bin/neo-go util upload-bin --cid 9iVfUg8aDHKjPC4LhQXEkVUM4HDkR7UCXYLs8NQwYfSG --wallet-config ./wallet-config.yml --block-attribute "ekt_eq" --index-attribute "oid-index" --rpc-endpoint https://rpc.t5.n3.nspcc.ru:20331 -fsr st1.t5.fs.neo.org:8080 -fsr st2.t5.fs.neo.org:8080 -fsr st3.t5.fs.neo.org:8080 -fsr st4.t5.fs.neo.org:8080 -fsr grpcs://st1.t5.fs.neo.org:8082

codecov · 2024-09-11T06:20:51Z

Codecov Report

Attention: Patch coverage is 12.25166% with 265 lines in your changes missing coverage. Please review.

Project coverage is 84.67%. Comparing base (86ed214) to head (6199240).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
cli/util/uploader.go	7.47%	257 Missing and 3 partials ⚠️
cli/util/convert.go	88.88%	1 Missing and 1 partial ⚠️
pkg/services/oracle/neofs/neofs.go	0.00%	2 Missing ⚠️
pkg/services/blockfetcher/blockfetcher.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3582      +/-   ##
==========================================
- Coverage   85.15%   84.67%   -0.49%     
==========================================
  Files         333      334       +1     
  Lines       39022    39321     +299     
==========================================
+ Hits        33231    33294      +63     
- Misses       4221     4449     +228     
- Partials     1570     1578       +8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

AliceInHunterland · 2024-09-11T06:22:54Z

@AnnaShaleva what do you think about adding more commands to neo-go cli?
The current pipeline of updating blocks in the container is as follows:

dump-bin
dump-bin-put
dump-generate-index-file
uploading index files (not a command)

cli/server/dump_bin_put.go

cli/server/server.go

cli/server/dump_bin_put.go

cli/util/convert.go

cli/util/uploader.go

AnnaShaleva · 2024-10-03T10:27:09Z

cli/util/uploader.go

+				defer wg.Done()
+				defer func() { <-workerPool }()
+
+				oidBlock, err := searchBlockOID(ctx, clientSDK, account, containerID, blockAttributeKey, index)


It's still there; we can't use Search request for every block, it's too time-consuming, imagine we have to perform 128K Search requests.

Time taken for index file generation with searches: 17m3.919291917s
Time taken for index file generation with getHeader: 14m57.008241s

I've expected a bigger difference, btw. Still, that's it.

cli/util/uploader.go

AnnaShaleva · 2024-10-04T09:03:38Z

BTW, linter is failing.

AliceInHunterland · 2024-10-05T19:20:30Z

706 seconds / 1000 blocks = 0.706 seconds/block. 11m46s for 1k blocks with slicer version

394 seconds / 1000 blocks = 0.394 seconds/block. 6m34s for 1k blocks with pool (it still a month)

with something like:

numWorkers := 5
	blockChan := make(chan int, numWorkers)
	errChan := make(chan error, numWorkers)
	var wg sync.WaitGroup

	for i := 0; i < numWorkers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for blockIndex := range blockChan {
                              block, err := rpc.GetBlockByIndex(uint32(blockIndex))
                              ....}()
         }
	go func() {
		for blockIndex := maxBlockIndex + 1; blockIndex <= int(currentBlockHeight); blockIndex++ {
			blockChan <- blockIndex
		}
		close(blockChan)
	}()
	
	go func() {
		wg.Wait()
		close(errChan)
	}()

74 seconds / 1000 blocks = 0.074 seconds/block. 1m14s for 1k blocks (~5 days)
producer(4)-consumer(6) 52 seconds
producer(10)-consumer(20) 15 seconds,but:

[2024-10-06T01:47:40+03:00] Successfully uploaded block: 442000
panic: runtime error: index out of range [-1]

goroutine 247 [running]:
math/rand.(*rngSource).Uint64(...)
        math/rand/rng.go:249
math/rand.(*rngSource).Int63(0x1400048d6a8?)
        math/rand/rng.go:234 +0x98
math/rand.(*Rand).Int63(...)
        math/rand/rand.go:96
math/rand.(*Rand).Int31(...)
        math/rand/rand.go:110
math/rand.(*Rand).Int31n(0x14000040750?, 0x48d7a0?)
        math/rand/rand.go:142 +0x8c
math/rand.(*Rand).Intn(0x10?, 0x10?)
        math/rand/rand.go:183 +0x30
github.com/nspcc-dev/neofs-sdk-go/pool.(*sampler).Next(0x140001cf3c0)
        github.com/nspcc-dev/[email protected]/pool/sampler.go:63 +0x2c
github.com/nspcc-dev/neofs-sdk-go/pool.(*innerPool).connection(0x140001cf400)
        github.com/nspcc-dev/[email protected]/pool/pool.go:988 +0x1a0
github.com/nspcc-dev/neofs-sdk-go/pool.(*Pool).connection(0x60?)
        github.com/nspcc-dev/[email protected]/pool/pool.go:967 +0x58
github.com/nspcc-dev/neofs-sdk-go/pool.(*Pool).sdkClient(0xad8f60c33a130b3?)
        github.com/nspcc-dev/[email protected]/pool/pool.go:1031 +0x1c
github.com/nspcc-dev/neofs-sdk-go/pool.(*Pool).ObjectPutInit(0x1400048da88, {0x101e0dec0, 0x1027af240}, {0x0, 0x0, 0x1400422a2a0, {0x0, 0x0, 0x0}}, {0x101e0e400, ...}, ...)
        github.com/nspcc-dev/[email protected]/pool/object.go:35 +0xa4
github.com/nspcc-dev/neo-go/cli/util.uploadBlock({0x101e0dec0, 0x1027af240}, {{0x14000192180, 0x1, 0x1}, {0x101e0e400, 0x140003f53b0}, 0x140002fb150, 0x14000436a80, 0x1400004b910, ...}, ...)
        github.com/nspcc-dev/neo-go/cli/util/uploader.go:401 +0x11c
github.com/nspcc-dev/neo-go/cli/util.uploadBin.func3()
        github.com/nspcc-dev/neo-go/cli/util/uploader.go:162 +0x6bc
created by github.com/nspcc-dev/neo-go/cli/util.uploadBin in goroutine 1
        github.com/nspcc-dev/neo-go/cli/util/uploader.go:145 +0xb48
(base) ekaterinapavlova@MacBook-Air-4 neo-go %

./bin/neo-go util upload-bin --cid 9iVfUg8aDHKjPC4LhQXEkVUM4HDkR7UCXYLs8NQwYfSG --wallet-config ./wallet-config.yml --block-attribute "ekt_block_pool_pr_cons" --index-attribute "oid-index" --rpc-endpoint https://rpc.t5.n3.nspcc.ru:20331 -fsr st1.t5.fs.neo.org:8080 -fsr st2.t5.fs.neo.org:8080 -fsr st3.t5.fs.neo.org:8080 -fsr st4.t5.fs.neo.org:8080

I haven't managed to repeat panic. successfully uploaded 1447000 blocks to testnet.

cli/util/uploader.go

cli/util/convert.go

cli/util/uploader.go

AliceInHunterland · 2024-10-09T06:11:43Z

cli/util/uploader.go

+	producerWg.Add(numProducers)
+
+	// Retry function with exponential backoff
+	retry := func(action func() error) error {


Should we save it? Maybe yes? I like it. With it, timeouts are rare occasions.

Let's keep it. Could you please point to the part of code that has a lot of timeout failures?

2024/09/26 20:46:33 Failed to fetch block 413230: failed to fetch block 413230: Post "https://rpc.morph.fs.neo.org": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Upload error: failed to fetch block 219272: Post "https://rpc.t5.n3.nspcc.ru:20331": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I think this one

also from history, but with slicer:

failed to upload block 150079: failed to create slicer: network info: status: code = 1024 message = websocket: close 1006 (abnormal closure): unexpected EOF 2024/09/26 21:28:26 Failed to upload block 630944: failed to create slicer: network info: write request: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html" failed to fetch max block index from container: failed to search objects from 321129 to 421128: error during object IDs iteration: rpc error: code = Internal desc = unexpected EOF

cli/util/uploader.go

cli/util/convert.go

docs/neofs-blockstorage.md

AnnaShaleva

Very good indeed! A couple of logic fixes are still needed, but overall looks much better.

pkg/services/oracle/neofs/neofs.go

cli/util/uploader.go

go.sum

pkg/services/oracle/neofs/neofs.go

docs/neofs-blockstorage.md

cli/util/uploader.go

go.sum

AnnaShaleva

OK, that's almost it. I need the following:

After you fix the remaining issues, try the final script on the testnet. Try to upload both blocks and index files (at least one index file is required). Provide at least three NeoFS SN addresses to the script to check how the pool works.
Add a tiny section to docs/neofs-blockstorage.md about this command. Write what it does and examples on how to use it.

pkg/services/blockfetcher/blockfetcher.go

cli/util/uploader.go

AnnaShaleva · 2024-10-16T12:58:04Z

Tests / Check internal dependencies (pull_request) check is failing. Try to rebase onto fresh master before fixing it.

With ObjectSearchInitter ObjectSearch can can be done by both NeoFS SDK client and pool. Signed-off-by: Ekaterina Pavlova <[email protected]>

Signed-off-by: Ekaterina Pavlova <[email protected]>

AnnaShaleva

Otherwise LGTM.

docs/neofs-blockstorage.md

AnnaShaleva · 2024-10-17T07:58:44Z

Tests / Check internal dependencies (pull_request) check is failing.

Still failing.

cli/util/uploader.go

This command is used for keeping container with blocks for blockfetcher updated. Close #3578 Signed-off-by: Ekaterina Pavlova <[email protected]>

AnnaShaleva

Good. If any bugs are found during the script usage, we'll fix them in a separate PR.

AliceInHunterland force-pushed the block-fetcher-commands branch from 8e1d6b3 to 64f6c7a Compare September 13, 2024 20:59

AliceInHunterland force-pushed the block-fetcher-commands branch 6 times, most recently from 2938069 to 13e075a Compare September 27, 2024 10:09

AnnaShaleva reviewed Oct 1, 2024

View reviewed changes

AliceInHunterland force-pushed the block-fetcher-commands branch from 13e075a to a6dc319 Compare October 1, 2024 15:04

AliceInHunterland marked this pull request as ready for review October 1, 2024 15:06

AliceInHunterland requested a review from roman-khimov as a code owner October 1, 2024 15:06

AliceInHunterland requested a review from AnnaShaleva October 1, 2024 15:08

AliceInHunterland force-pushed the block-fetcher-commands branch from a6dc319 to e5601ee Compare October 1, 2024 15:09

AnnaShaleva reviewed Oct 3, 2024

View reviewed changes

cli/util/uploader.go Outdated Show resolved Hide resolved

AliceInHunterland force-pushed the block-fetcher-commands branch from e5601ee to 1ddbbda Compare October 5, 2024 19:21

AliceInHunterland commented Oct 6, 2024

View reviewed changes

cli/util/uploader.go Outdated Show resolved Hide resolved

AliceInHunterland requested a review from AnnaShaleva October 7, 2024 07:15

AliceInHunterland mentioned this pull request Oct 7, 2024

Pool unexpected panic nspcc-dev/neofs-sdk-go#631

Open

AnnaShaleva reviewed Oct 7, 2024

View reviewed changes

AliceInHunterland force-pushed the block-fetcher-commands branch from 1ddbbda to 0a4e1d1 Compare October 9, 2024 06:09

AliceInHunterland commented Oct 9, 2024

View reviewed changes

cli/util/uploader.go Outdated Show resolved Hide resolved

AliceInHunterland commented Oct 9, 2024

View reviewed changes

cli/util/convert.go Show resolved Hide resolved

AliceInHunterland commented Oct 9, 2024

View reviewed changes

docs/neofs-blockstorage.md Show resolved Hide resolved

AliceInHunterland requested a review from AnnaShaleva October 9, 2024 06:39

AnnaShaleva reviewed Oct 9, 2024

View reviewed changes

AliceInHunterland force-pushed the block-fetcher-commands branch 2 times, most recently from b864e98 to 110d45b Compare October 14, 2024 22:25

AliceInHunterland requested a review from AnnaShaleva October 15, 2024 07:10

AnnaShaleva reviewed Oct 15, 2024

View reviewed changes

AliceInHunterland mentioned this pull request Oct 15, 2024

neofs.ObjectSearch returned fewer objects than expected #3615

Open

AliceInHunterland force-pushed the block-fetcher-commands branch 2 times, most recently from 2166582 to be76869 Compare October 15, 2024 13:37

AnnaShaleva reviewed Oct 16, 2024

View reviewed changes

AliceInHunterland added 3 commits October 16, 2024 19:23

neofs: add ObjectSearchInitter

2435484

With ObjectSearchInitter ObjectSearch can can be done by both NeoFS SDK client and pool. Signed-off-by: Ekaterina Pavlova <[email protected]>

config, docs: adjust NeoFSBlockFetcher index file and time attributes

852dcb0

Signed-off-by: Ekaterina Pavlova <[email protected]>

blockfetcher: adjust index file numeration from 0

59fab5d

Signed-off-by: Ekaterina Pavlova <[email protected]>

AliceInHunterland force-pushed the block-fetcher-commands branch from be76869 to 1313199 Compare October 16, 2024 16:24

AnnaShaleva reviewed Oct 17, 2024

View reviewed changes

docs/neofs-blockstorage.md Outdated Show resolved Hide resolved

docs/neofs-blockstorage.md Outdated Show resolved Hide resolved

docs/neofs-blockstorage.md Outdated Show resolved Hide resolved

docs/neofs-blockstorage.md Outdated Show resolved Hide resolved

AnnaShaleva reviewed Oct 17, 2024

View reviewed changes

cli/util/uploader.go Show resolved Hide resolved

AliceInHunterland force-pushed the block-fetcher-commands branch from 1313199 to dfbd127 Compare October 17, 2024 08:40

cli: add upload-bin

6199240

This command is used for keeping container with blocks for blockfetcher updated. Close #3578 Signed-off-by: Ekaterina Pavlova <[email protected]>

AliceInHunterland force-pushed the block-fetcher-commands branch from dfbd127 to 6199240 Compare October 17, 2024 08:56

AnnaShaleva approved these changes Oct 17, 2024

View reviewed changes

AnnaShaleva merged commit 14ea5a8 into master Oct 17, 2024
33 of 35 checks passed

AnnaShaleva deleted the block-fetcher-commands branch October 17, 2024 09:10

AnnaShaleva mentioned this pull request Oct 17, 2024

BlockUploader fails due to timeouts #3620

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NeoFS block storage: add uploading commands #3582

NeoFS block storage: add uploading commands #3582

AliceInHunterland commented Sep 11, 2024 •

edited

Loading

codecov bot commented Sep 11, 2024 •

edited

Loading

AliceInHunterland commented Sep 11, 2024

AnnaShaleva Oct 3, 2024

AliceInHunterland Oct 4, 2024

roman-khimov Oct 4, 2024

AnnaShaleva commented Oct 4, 2024

AliceInHunterland commented Oct 5, 2024 •

edited

Loading

AliceInHunterland Oct 9, 2024 •

edited

Loading

AnnaShaleva Oct 9, 2024

AliceInHunterland Oct 14, 2024

AnnaShaleva left a comment

AnnaShaleva left a comment

AnnaShaleva commented Oct 16, 2024

AnnaShaleva left a comment

AnnaShaleva commented Oct 17, 2024

AnnaShaleva left a comment

NeoFS block storage: add uploading commands #3582

NeoFS block storage: add uploading commands #3582

Conversation

AliceInHunterland commented Sep 11, 2024 • edited Loading

codecov bot commented Sep 11, 2024 • edited Loading

Codecov Report

AliceInHunterland commented Sep 11, 2024

AnnaShaleva Oct 3, 2024

Choose a reason for hiding this comment

AliceInHunterland Oct 4, 2024

Choose a reason for hiding this comment

roman-khimov Oct 4, 2024

Choose a reason for hiding this comment

AnnaShaleva commented Oct 4, 2024

AliceInHunterland commented Oct 5, 2024 • edited Loading

AliceInHunterland Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

AnnaShaleva Oct 9, 2024

Choose a reason for hiding this comment

AliceInHunterland Oct 14, 2024

Choose a reason for hiding this comment

AnnaShaleva left a comment

Choose a reason for hiding this comment

AnnaShaleva left a comment

Choose a reason for hiding this comment

AnnaShaleva commented Oct 16, 2024

AnnaShaleva left a comment

Choose a reason for hiding this comment

AnnaShaleva commented Oct 17, 2024

AnnaShaleva left a comment

Choose a reason for hiding this comment

AliceInHunterland commented Sep 11, 2024 •

edited

Loading

codecov bot commented Sep 11, 2024 •

edited

Loading

AliceInHunterland commented Oct 5, 2024 •

edited

Loading

AliceInHunterland Oct 9, 2024 •

edited

Loading