Skip to content

Commit

Permalink
upload chunks in parallel (#1376)
Browse files Browse the repository at this point in the history
* Update sprint 1.12 (#1341)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix trailing whitespace (#1343)

* Merge staging changes (#1346)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

---------

Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix upload select (#1351)

* fix workdir in mobile sdk (#1345)

* fix mobile workdir

* set multi op batch size

* set option to download to disk in wasm (#1348)

* fix panic in hash chan (#1352)

* Fix merge conflict in sprint-1.12 (#1354)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* repair in batches (#1347)

* repair in batches

* fix lint

* fix unit test

* fix batch size

---------

Co-authored-by: Yury <[email protected]>

* Merge staging (#1365)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

* Sprint changes (#1355)

* Update sprint 1.12 (#1341)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix trailing whitespace (#1343)

* Merge staging changes (#1346)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

---------

Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix upload select (#1351)

* fix workdir in mobile sdk (#1345)

* fix mobile workdir

* set multi op batch size

* set option to download to disk in wasm (#1348)

* fix panic in hash chan (#1352)

* Fix merge conflict in sprint-1.12 (#1354)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* repair in batches (#1347)

* repair in batches

* fix lint

* fix unit test

* fix batch size

---------

Co-authored-by: Yury <[email protected]>

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Yury <[email protected]>

* fix repair after update (#1357)

* fix err msg (#1361)

* fix err msg

* calc root hash once

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Yury <[email protected]>

* Fix stake pool stats (#1356)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

* Sprint changes (#1355)

* Update sprint 1.12 (#1341)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix trailing whitespace (#1343)

* Merge staging changes (#1346)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

---------

Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix upload select (#1351)

* fix workdir in mobile sdk (#1345)

* fix mobile workdir

* set multi op batch size

* set option to download to disk in wasm (#1348)

* fix panic in hash chan (#1352)

* Fix merge conflict in sprint-1.12 (#1354)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* repair in batches (#1347)

* repair in batches

* fix lint

* fix unit test

* fix batch size

---------

Co-authored-by: Yury <[email protected]>

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Yury <[email protected]>

* Fix stake pool stats

---------

Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Ebrahim Gomaa <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Yury <[email protected]>

* list pagination (#1368)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

* add pagination in list

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* Fix sync in windows (#1370)

* Add option for mimeType (#1372)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

* add mimeType as option

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>
Co-authored-by: Yury <[email protected]>

* fix shutdown blobber (#1374)

(cherry picked from commit 0fbb702)

* Cancel upload (#1332)

* feature: added more test cases

* fix: fixed existing test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added more test cases

* feature: added more test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added mocks

* fix: fixed bugs

* fix: fixed mocks

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: made components internal

* Add webstreaming to multiupload (#1190)

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* fix dup upload consensus (#1195)

* Fix multiupload completed callback (#1172)

* remove unnecessary print

* fix delete and createdir

* return major error

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* cleanup

* fix completed callback

* Feat/update methods in WinSDK (#1198)

* delete dir in repair (#1196)

* delete dir in repair

* replace ioutil

* fix unit test

* fix list worker test

* fix rename dir error

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* Add NFT Config contract (#1212)

* Silent the zcnbridge http request when --silent (#1207)

* Change the return type of downloadBlocks to []byte in wasm (#1218)

* change the order of exchanges to get the ZCN prices (#1216)

* Deprecate non multi-operation functions (#1214)

* remove CreateDir, Rename, Copy, Move

* use docker build wasm

* use @v4 version of git action

* Revert "use @v4 version of git action"

This reverts commit fbb93f2.

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

* remove path header, fix ws tag (#1220)

* Feature/ Challenge based on rounds (#1191)

* Fix

* Fix

* Fix

* Fix

* Fix

* Debug

* Debug

* Debug

* Fix

* Heavy logging

* Fix

* More logging

* Changed logging

* Removed logging

* Debug

* Fix

* Fix

* Fix

* expose send in wasm (#1225)

* fix rename dir err (#1226)

* Txnfee send (#1227)

* add txnfee for send function

* return txn verify output

* add rename dir (#1230)

* Async read (#1213)

* basic timings for download

* upload timings

* to seconds

* more timings for upload processing

* display ms properly

* cleanup

* fix merge

* fix merge

* add more timing logs

* fix dur to ms

* add timings to read and build data

* async read

* range over errChan

* fix unit test

* rmv ctx cancel from multi upload

* wait for last chunk

* fix loop

* rmv readChunk timing

* increase buffer size

* rmv timing logs

* cleanup

* updated go to version 1.20

* use -buildvcs=false

---------

Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>

* Wasm for `move`, `rename`, `copy` (#1233)

* add demo for move, rename and copy

* remove debug logs

---------

Co-authored-by: Yury <[email protected]>

* Feat/streaming server in winsdk (#1228)

* fix move when srcPath is same as destPath (#1239)

* fix(upload):fixed invalid ffmpeg command in transcode feature (#1236)

* fix(upload):fixed invalid ffmpeg command in transcode feature

* fix(upload): always delete transcode output file

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(upload): fixed HideWindow issue

* fix(devops): fixed build-windows

* fix(mobilesdk): add logging for MultiUpload

* fix(mobilesdk): fixed file permission issue on transcode

* fixed lint error

---------

Co-authored-by: dabasov <[email protected]>

* Fix web streaming - wasm (#1237)

* Do not send GET http request with body

* Use downloadBlocks to download in player_file

* Call ListDir to get file meta

* Format

* Download 100 blocks each time by default

* Fix downloadBlocks

Track the downloaded bytes and trunk the padding 0 bytes

* Update downloadBlocks to download blocks concurretly

Use the same download method with downloadFileByBlocks

* Add ActualFileSize in PlayListFile

* Update wasm_exec.js to go 1.21.0

* Update version

* Replace alloc.DownloadBlocks with DownloadBlocksToFileHandler

* Add concurrency control (#1245)

* control file ops concurrency

* cleanup

* add batch size to multi op

* Fix/stream tests (#1247)

* fix(upload): fixed file name issue in webstreaming

* removed unused code

* fix share consensus (#1246)

* List optimization (#1240)

* list optimization

* list optimization

* fix list hash

* add to wg

* fix list test

* check consensus

* fix listDir test

* use once for mock calls

* added sharders keep list (#1231)

* added sharders keep list

* added holder to zboxcore

* removed duplicated sharder call

* fixed lint

* fixed panic

* fixed panic

* fixed panic

* fixed panic

* fixed panic

* added consensus

* added consensus

* added consensus

* refactored

* refactored

* refactored

* chec alloc and prev alloc root (#1250)

* add check for chunkNumber (#1253)

* add check for chunkNumber

* add err log

* fix save progress

* split-key handler implemented in wasm sdk

* chnaging go version for cdn

* bumped version

* minor refactoring

* bumping  wasm_exec.js  version

* Refactoring unnecessary structs

* create new conID (#1256)

* debugging consoles added

* fix statusbar update (#1257)

* lint fixes

* lint ignore

* nolint: unused added

* use httpdo for create connection (#1259)

* Add create dir in repair (#1262)

* fix delete dir err in cb

* add way to create dir in repair

* add desc for transaction data (#1263)

* Feature: provide audit zcnswap and refactor if needed (#1205)

* feature: modifies zcnswap to be consitent with zcnbridge

* fix: fixed bugs

* fix: removed dead code

* feature: extended key store logic

* fix: fixed bugs

* fix: fixed conflicts

* fix: removed dead code

* fix: fixed bugs

* Fix/remove path (#1221)

* feature: added more test cases

* fix: fixed existing test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added more test cases

* feature: added more test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added mocks

* fix: fixed bugs

* fix: fixed mocks

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: made components internal

* Add webstreaming to multiupload (#1190)

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* fix dup upload consensus (#1195)

* Fix multiupload completed callback (#1172)

* remove unnecessary print

* fix delete and createdir

* return major error

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* cleanup

* fix completed callback

* Feat/update methods in WinSDK (#1198)

* delete dir in repair (#1196)

* delete dir in repair

* replace ioutil

* fix unit test

* fix list worker test

* fix rename dir error

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* Add NFT Config contract (#1212)

* Silent the zcnbridge http request when --silent (#1207)

* Change the return type of downloadBlocks to []byte in wasm (#1218)

* change the order of exchanges to get the ZCN prices (#1216)

* Deprecate non multi-operation functions (#1214)

* remove CreateDir, Rename, Copy, Move

* use docker build wasm

* use @v4 version of git action

* Revert "use @v4 version of git action"

This reverts commit fbb93f2.

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

* remove path header, fix ws tag

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>

* Revert "Fix/remove path (#1221)" (#1222)

This reverts commit 464260a.

* remove path header, fix ws tag (#1223)

* remove path header, fix ws tag

* fixed broken build

---------

Co-authored-by: din-mukhammed <[email protected]>

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

(cherry picked from commit 1c9f143)

* fix

* updated wasm binary build fix.

* feature: added test case for swap logic

* fix: resolved dependency conflicts

* fix: fixed bugs

* feature: added zcn eth rate fetch logic

* Update blobber.go

* Update authorizers_query.go

* feature: replaced confusing param naming

* fix: updated bridge initialization in wasm bridge integration

* fix: replaced value amount with correct value

* fix: wrong test case values

* feature: switch to the usage of zcn token smart contract

* fix: fixed bridge tests

---------

Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Yury <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>

* Feat/exposed cli methods in winsdk (#1248)

* feat(winsdk): added CreateWallet

* feat(winsdk): added RecoverWallet

* feat(winsdk): added logging in CreateWallet and RecoverWallet

* feat(winsdk): split InitSDK into InitZCNSDK and InitWallets

* feat(winsdk): fixed zboxApiClient initialization issue

* feat(winsdk): renamed init sdks

* feat(winsdk): fixed init zboxApiClient issue

* fix(winsdk): fixed RecoverWallet naming

* fix(winsdk): fixed file name issue in GetUploadStatus for transcode feature

* feat(winsdk): added GetFileContentType in winsdk

* feat(winsdk): added ListAllocations

* feat(winsdk): added CreateFreeAllocation

* feat(winsdk): added allocation methods

* fix(winsdk): added logs

* feat(zboxapi): added GetFreeStorage

* feat(zboxapi): also send X-App-ID-Token in header

* fix(winsdk): skip TestGetFreeStorage

* fix(winsdk): added getAllocationWith

* fix(zboxapi): used local signHash instead of sys.SignFunc

* fix(zboxapi): fixed token issue in GetFreeStorage

* fix(list): exported ThumbnailHash and ThumbnailSize in ListResult

* fix(0box): fixed GetFreeStorage/CreateFreeAllocation

* fix(winsdk): renamed jwtToken with token in CreateFreeAllocation

* fix(winsdk): renamed jwtToken with token in CreateFreeAllocation

* feat(winsdk): added GetFreeMarker

* feat(winsdk): added AddSharedInfo/DeleteSharedInfo/GetSharedToMe/GetSharedByMe

* feat(winsdk): added CreateAuthTicket/DeleteAuthTicket

* feat(winsdk): fixed GetSharedToMe/GetSharedByMe

* feat(winsdk): fixed GetSharedToMe/GetSharedByMe

* feat(winsdk): exported CreateAuthTicket/DeleteAuthTicket/CreateSharedInfo/DeleteSharedInfo/GetSharedByMe/GetSharedByPublic/GetSharedToMe

* Fix/fail shardedr on bad code (#1264)

* Fix/remove path (#1221)

* feature: added more test cases

* fix: fixed existing test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added more test cases

* feature: added more test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added mocks

* fix: fixed bugs

* fix: fixed mocks

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: made components internal

* Add webstreaming to multiupload (#1190)

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* fix dup upload consensus (#1195)

* Fix multiupload completed callback (#1172)

* remove unnecessary print

* fix delete and createdir

* return major error

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* cleanup

* fix completed callback

* Feat/update methods in WinSDK (#1198)

* delete dir in repair (#1196)

* delete dir in repair

* replace ioutil

* fix unit test

* fix list worker test

* fix rename dir error

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* Add NFT Config contract (#1212)

* Silent the zcnbridge http request when --silent (#1207)

* Change the return type of downloadBlocks to []byte in wasm (#1218)

* change the order of exchanges to get the ZCN prices (#1216)

* Deprecate non multi-operation functions (#1214)

* remove CreateDir, Rename, Copy, Move

* use docker build wasm

* use @v4 version of git action

* Revert "use @v4 version of git action"

This reverts commit fbb93f2.

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

* remove path header, fix ws tag

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>

* Revert "Fix/remove path (#1221)" (#1222)

This reverts commit 464260a.

* remove path header, fix ws tag (#1223)

* remove path header, fix ws tag

* fixed broken build

---------

Co-authored-by: din-mukhammed <[email protected]>

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

(cherry picked from commit 1c9f143)

* updated wasm binary build fix.

* README grammar and formatting tweaks.

* Merge remote-tracking branch 'origin/feature/sharder-keep-list' into feature/sharder-keep-list

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: UncertainBadg3r <[email protected]>

* Fix/roll logs (#1266)

* Added lumberjack logger

* Added lumberjack logger

* fix:download file/thumbnail by authticket in winsdk (#1265)

* feat(winsdk): added GetWalletBalance (#1267)

* enabled gosdk update for sprint branches (#1268)

* Revert "enabled gosdk update for sprint branches (#1268)" (#1269)

This reverts commit 201b8bd.

* fix in progress callback (#1270)

* Cleanup update terms from update allocation (#1274)

* Improve download performance (#1271)

* improve download performance

* add read buffer size

* add hash option

* fix lint test

* set default to true

* log for data

* fix read

* feature: add reset nonce for bridge (#1279)

* Add registerAuthorizer to wasm (#1275)

* Hotfix/parse hostname error (#1241)

* Fix/remove path (#1221)

* feature: added more test cases

* fix: fixed existing test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added more test cases

* feature: added more test cases

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* feature: added mocks

* fix: fixed bugs

* fix: fixed mocks

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: made components internal

* Add webstreaming to multiupload (#1190)

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* fix dup upload consensus (#1195)

* Fix multiupload completed callback (#1172)

* remove unnecessary print

* fix delete and createdir

* return major error

* add webstreaming

* fix typos

* add webstreaming to wasm multiupload

* fix typo

* cleanup

* fix completed callback

* Feat/update methods in WinSDK (#1198)

* delete dir in repair (#1196)

* delete dir in repair

* replace ioutil

* fix unit test

* fix list worker test

* fix rename dir error

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* fix: fixed bugs

* Add NFT Config contract (#1212)

* Silent the zcnbridge http request when --silent (#1207)

* Change the return type of downloadBlocks to []byte in wasm (#1218)

* change the order of exchanges to get the ZCN prices (#1216)

* Deprecate non multi-operation functions (#1214)

* remove CreateDir, Rename, Copy, Move

* use docker build wasm

* use @v4 version of git action

* Revert "use @v4 version of git action"

This reverts commit fbb93f2.

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

* remove path header, fix ws tag

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>

* Revert "Fix/remove path (#1221)" (#1222)

This reverts commit 464260a.

* remove path header, fix ws tag

* remove path header, fix ws tag (#1223)

* remove path header, fix ws tag

* fixed broken build

---------

Co-authored-by: din-mukhammed <[email protected]>

* Updated self hosted macos runner (#1219)

* Updated self hosted macos runner

* fixed runner name build-sdks.yml

(cherry picked from commit 1c9f143)

* added logging

* updated wasm binary build fix.

* updated wasm binary build fix.

(cherry picked from commit 2571888)

* README grammar and formatting tweaks.

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: UncertainBadg3r <[email protected]>
Co-authored-by: Jayash Satolia <[email protected]>

* Adding RegisterAuthorizer method in wasmsdk (#1273)

* Adding RegisterAuthorizer method in wasmsdk

* Fix:wasm test fail

* Adding demo for RegisterAuthorizer handler in wasm

* Js and Go bridge implementation

* RegisterAuthorizer handler changes and adding Async Js bridge

* golangci-lint fixes

* Added error handling in setSplitKey and setAuthURL

* Refactoring wasm exposed handlers

* Setting split key wallet before setting auth url in setAuthURL's demo

* Lint Fixes after resolving conflicts

* upload optimization (#1281)

* add sha256 simd and blake3

* add hash chan

* fix lint and unit test

* fix MHash

* rmv timing log

* fix thumbnail hash

* added logging

* added logging

* hotfix

* hotfix

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* experiment

* Sprint 1.11 revert (#1284)

* Revert "experiment"

This reverts commit 050270f.

* Revert "experiment"

This reverts commit d4821f8.

* Revert "experiment"

This reverts commit 9f65b2e.

* Revert "experiment"

This reverts commit 6313cb8.

* Revert "experiment"

This reverts commit b6be0b5.

* Revert "experiment"

This reverts commit 68d4190.

* Revert "experiment"

This reverts commit 3a23f3f.

* Revert "experiment"

This reverts commit 6b6578b.

* Revert "experiment"

This reverts commit 65a0152.

* Revert "experiment"

This reverts commit 6834b05.

* Revert "experiment"

This reverts commit 8446894.

* Revert "hotfix"

This reverts commit 48cf793.

* Revert "hotfix"

This reverts commit 895c7f0.

* Revert "added logging"

This reverts commit fafce3c.

* Revert "added logging"

This reverts commit 10b6a4a.

* rmv channel hasher (#1285)

* Fix submit txn (#1287)

* Feature: add BNT, USDC and EURC tokens for Swap (#1277)

* feature: add multiple token support for swap

* feature: add correct Bancor API calls

* feature: added approve for swap of bancor token

* fix: removed debug lines

* fix: fixed swap approval

* fix: fixed bug

* fix: fixed bugs

* change hasher (#1288)

* change hasher

* set max proc to 1

* set batch size to 2

* revert hasher lib changes

* increase batch size to 4

* set batch size to 2

* fix build

* update chunk reader

* run gc

* add sleep and decrease batch size

* change blake3

* fix operation slice

* fix batch size

* increase sleep time

* add memory pool

* increase swg for wasm

* reuse chunk reader

* fix read chunk

* rmv unused package

* Remove sleep and GOMAXPROCS=1

* Increase batch size a little

---------

Co-authored-by: peterlimg <[email protected]>

* Import eth wallet with index (#1292)

* Add import account address index

* Make it optional

* Feat/import-account-idx (#1294)

* Add import account address index

* Make it optional

* Add account index option

* Add account bip32 option (#1295)

* Fix allocation min lock (#1297)

* create PRs to dependent repos when a PR is merged to sprint branches (#1283)

* create PRs to dependent repos when a PR is merged to sprint branches

* add branch name

* move wasm-build to build-sdk.yml

* cleanup sdk-release.yml

* Removed padding from the CryptoJS Encrypt and Decrypt functions functions. (#1300)

Co-authored-by: Manohar Reddy <[email protected]>

* Fix allocation min lock

* change chunkNumber for wasm (#1304)

* change numBlock for wasm

* change numBlocks

* revert sdk changes

* Feature: implement fork per run strategy (#1299)

* feature: added fork creation

* fix: fixed bug

* feature: improved Tenderly fork creation flow

* fix parse error (#1309)

* fix parse error

* fix log

* add swg

* Fix memory usage (#1311)

* fix memory usage

* change hasher to sha2 256

* fix unit test

* increase numBlocks

* fix thumbnail hash

* Fix iso header (#1314)

* add ISO check

* remove filename field from authticket

* fix typo

* log auth token

* cleanup

* Fix zboxcore min submit (#1315)

* Fix zboxcore min submit

* Fix min submit and min confirmation default configs

* Fix unit tests

* Fix unit tests

* Revert "Fix iso header (#1314)" (#1317)

This reverts commit 218d3ed.

* Add upload for empty file (#1316)

* add upload for empty file

* fix hash

* check ref

* fix actual size

* fix empty upload (#1320)

* fix read auth ticket (#1319)

* allocation validation for data parity shards (#1321)

* fix git action to raise PRs to update gosdk (#1313)

* test gosdk update

* test-gosdk-update

* use binary operators instead of terinary

* use github.ref

* use github ref

* add branch name

* use quotes

* prepare for PR

* test work flow dispatch

* test extract branch

* print env

* prepare for PR

* feature: added custom block number during Tenderly fork creation (#1323)

* Fix/retry tx on nonce error (#1322)

* Revert "experiment"

This reverts commit 050270f.

* Revert "experiment"

This reverts commit d4821f8.

* Revert "experiment"

This reverts commit 9f65b2e.

* Revert "experiment"

This reverts commit 6313cb8.

* Revert "experiment"

This reverts commit b6be0b5.

* Revert "experiment"

This reverts commit 68d4190.

* Revert "experiment"

This reverts commit 3a23f3f.

* Revert "experiment"

This reverts commit 6b6578b.

* Revert "experiment"

This reverts commit 65a0152.

* Revert "experiment"

This reverts commit 6834b05.

* Revert "experiment"

This reverts commit 8446894.

* Revert "hotfix"

This reverts commit 48cf793.

* Revert "hotfix"

This reverts commit 895c7f0.

* Revert "added logging"

This reverts commit fafce3c.

* Revert "added logging"

This reverts commit 10b6a4a.

* retry failed transaction in zbox with nonce error

* Fix wm rollback (#1324)

* fix wm rollback

* fix rb check

* version logs

* fix prev version

* cleanup

* Cleanup free storage update allocation (#1325)

* remove batch size (#1327)

* fix retry op callback (#1329)

* Revert "remove batch size (#1327)" (#1328)

This reverts commit 310edbb.

* fix cancel upload

* Add `force` param for getAllocationBlobbers wasm (#1330)

* fix retry callback (#1331)

* Fix fill panic (#1334)

* fix fill shards panic

* add log for fill err

* fix err

* add cancel upload to wasm

* fix proxy

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

* Sprint changes (#1355)

* Update sprint 1.12 (#1341)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix trailing whitespace (#1343)

* Merge staging changes (#1346)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

---------

Co-authored-by: Hitenjain14 <[email protected]>
Co-authored-by: Ebrahim Gomaa <[email protected]>

* fix upload select (#1351)

* fix workdir in mobile sdk (#1345)

* fix mobile workdir

* set multi op batch size

* set option to download to disk in wasm (#1348)

* fix panic in hash chan (#1352)

* Fix merge conflict in sprint-1.12 (#1354)

* wait for repair and increase numBlocks (#1338)

* reorder wait group done (#1340)

* hotfix / remove hard coded prefix handling of encrypted upload (#1344)

* remove hard coded prefix handling of encrypted upload

* fix for other parts of the file

* increase batch size (#1349)

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>

* repair in batches (#1347)

* repair in batches

* fix lint

* fix unit test

* fix batch size

---------

Co-authored-by: Yury <[email protected]>

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Yury <[email protected]>

* fix repair after update (#1357)

* use mo ctx as parent

* fix wasm build

* empty commit

* fix ctx

---------

Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: Jayash Satolia <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: Yury <[email protected]>
Co-authored-by: Ash-KODES <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: UncertainBadg3r <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Tanishq Gupta <[email protected]>
Co-authored-by: Jayash Satolia <[email protected]>
Co-authored-by: GuruHubb <[email protected]>
Co-authored-by: Ebrahim Gomaa <[email protected]>

* Fix auth response on wasm (#1359)

* Fix auth response on wasm

* Fix verifyHandler

* Fix auth sign wasm

* Update pkg for gomobile

* Remove test code

* fixed lint

---------

Co-authored-by: dabasov <[email protected]>

* upload chunks in parallel

* fix upload wg

* Download buffer (#1363)

* revert index changes

* revert close changes

* rmv data buffer

* download buffer

* fix write to file

* rmv len check

* fix decodeEC ut

* fix remaining size

* cleanup

* fix mem chan file

* fix dataBuffers

* add logs and dec workers

* discard body is status ok

* set idle connections

* fix lint

* fix content type header

* fix req complete cb (#1381)

* set blocks to 60 and add fasthttp

* set updated in progress

* rmv in load

* fix lint

* add mock for fasthttp

* cleanup

* add worker support

* fix import

* rmv alreadyUploaded

* get hardfork round by name (#1385)

* get hardfork round by name

* fix naming

* fix default hardfork round

* made GetFreeAllocationBlobbers public

* set fs next

* fixed resp format

* fixed resp format

* [wasm] Expose minersc - stakePool/unstakePool collect rewards (#1386)

* replaced coingecko url with internal price url (#1379)

* Add miner/sharder stake and unstake support

* Add collect rewards for miner/sharder

---------

Co-authored-by: Gaurang Patel <[email protected]>

* fix lint false positive

* upload chunks in parallel

* fix upload wg

* fix dataBuffers

* add logs and dec workers

* discard body is status ok

* set idle connections

* fix lint

* fix content type header

* set blocks to 60 and add fasthttp

* set updated in progress

* rmv in load

* fix lint

* add mock for fasthttp

* cleanup

* add worker support

* fix import

* rmv alreadyUploaded

* fix lint false positive

* set fs next

* add timer

* add timer

---------

Co-authored-by: Ebrahim Gomaa <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Yury <[email protected]>
Co-authored-by: Jayash Satolia <[email protected]>
Co-authored-by: Roshan Kumar <[email protected]>
Co-authored-by: Manohar Reddy <[email protected]>
Co-authored-by: YarikRevich <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Dinmukhammed Kambarov <[email protected]>
Co-authored-by: Lz <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: Kishan Dhakan <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: din-mukhammed <[email protected]>
Co-authored-by: shahnawaz-creator <[email protected]>
Co-authored-by: Ash-KODES <[email protected]>
Co-authored-by: Yaroslav Svitlytskyi <[email protected]>
Co-authored-by: UncertainBadg3r <[email protected]>
Co-authored-by: peterlimg <[email protected]>
Co-authored-by: Tanishq Gupta <[email protected]>
Co-authored-by: Jayash Satolia <[email protected]>
Co-authored-by: GuruHubb <[email protected]>
Co-authored-by: Gaurang Patel <[email protected]>
  • Loading branch information
24 people authored Feb 22, 2024
1 parent 18aad7b commit acaf667
Show file tree
Hide file tree
Showing 16 changed files with 658 additions and 319 deletions.
8 changes: 6 additions & 2 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ require (
github.com/uptrace/bunrouter v1.0.20
go.dedis.ch/kyber/v3 v3.1.0
go.uber.org/zap v1.24.0
golang.org/x/crypto v0.16.0
golang.org/x/crypto v0.17.0
golang.org/x/image v0.14.0
golang.org/x/sync v0.5.0
google.golang.org/grpc v1.53.0
Expand All @@ -45,10 +45,12 @@ require (
require (
github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d
github.com/minio/sha256-simd v1.0.1
github.com/valyala/fasthttp v1.51.0
)

require (
github.com/VictoriaMetrics/fastcache v1.6.0 // indirect
github.com/andybalholm/brotli v1.0.5 // indirect
github.com/btcsuite/btcd/btcec/v2 v2.3.2 // indirect
github.com/btcsuite/btcd/chaincfg/chainhash v1.0.2 // indirect
github.com/cespare/xxhash v1.1.0 // indirect
Expand Down Expand Up @@ -112,6 +114,7 @@ require (
github.com/tklauser/go-sysconf v0.3.11 // indirect
github.com/tklauser/numcpus v0.6.0 // indirect
github.com/urfave/cli/v2 v2.17.2-0.20221006022127-8f469abc00aa // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
go.dedis.ch/fixbuf v1.0.3 // indirect
go.opencensus.io v0.24.0 // indirect
Expand All @@ -128,7 +131,8 @@ require (

require (
github.com/btcsuite/btcd/btcutil v1.1.3
github.com/klauspost/compress v1.16.0 // indirect
github.com/hitenjain14/fasthttp v0.0.0-20240201092245-8e4835c0e974
github.com/klauspost/compress v1.17.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.4 // indirect
github.com/remeh/sizedwaitgroup v1.0.0
github.com/yusufpapurcu/wmi v1.2.2 // indirect
Expand Down
16 changes: 12 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRF
github.com/allegro/bigcache v1.2.1-0.20190218064605-e24eb225f156 h1:eMwmnE/GDgah4HI848JfFxHt+iPb26b4zyfspmqY0/8=
github.com/allegro/bigcache v1.2.1-0.20190218064605-e24eb225f156/go.mod h1:Cb/ax3seSYIx7SuZdm2G2xzfwmv3TPSk2ucNfQESPXM=
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
github.com/andybalholm/brotli v1.0.5 h1:8uQZIdzKmjc/iuPu7O2ioW48L81FgatrcpfFmiq/cCs=
github.com/andybalholm/brotli v1.0.5/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/apache/arrow/go/arrow v0.0.0-20191024131854-af6fa24be0db/go.mod h1:VTxUBvSJ3s3eHAg65PNgrsn5BtqCRPdmyXh6rAfdxN0=
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8=
Expand Down Expand Up @@ -297,6 +299,8 @@ github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/herumi/bls-go-binary v1.31.0 h1:L1goQ2tMtGgpXCg5AwHAdJQpLs/pfnWWEc3Wog6OhmI=
github.com/herumi/bls-go-binary v1.31.0/go.mod h1:O4Vp1AfR4raRGwFeQpr9X/PQtncEicMoOe6BQt1oX0Y=
github.com/hitenjain14/fasthttp v0.0.0-20240201092245-8e4835c0e974 h1:oEjH9SSKBlzwDyYjzZaqRpxo7GlfUJCyRoOk7QHKSDs=
github.com/hitenjain14/fasthttp v0.0.0-20240201092245-8e4835c0e974/go.mod h1:RZMcXy7u4S+E97IXYTe7WHZ3+mCYOh4vys8PkIGZeXk=
github.com/holiman/bloomfilter/v2 v2.0.3 h1:73e0e/V0tCydx14a0SCYS/EWCxgwLZ18CZcZKVu0fao=
github.com/holiman/bloomfilter/v2 v2.0.3/go.mod h1:zpoh+gs7qcpqrHr3dB55AMiJwo0iURXE7ZOP9L9hSkA=
github.com/holiman/uint256 v1.2.2-0.20230321075855-87b91420868c h1:DZfsyhDK1hnSS5lH8l+JggqzEleHteTYfutAiVlSUM8=
Expand Down Expand Up @@ -336,8 +340,8 @@ github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+o
github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4=
github.com/klauspost/compress v1.4.0/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg=
github.com/klauspost/compress v1.16.0 h1:iULayQNOReoYUe+1qtKOqw9CwJv3aNQu8ivo7lw1HU4=
github.com/klauspost/compress v1.16.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/klauspost/compress v1.17.0 h1:Rnbp4K9EjcDuVuHtd0dgA4qNuv9yKDYKK1ulpJwgrqM=
github.com/klauspost/compress v1.17.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/klauspost/cpuid v0.0.0-20170728055534-ae7887de9fa5/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/klauspost/cpuid/v2 v2.2.4 h1:acbojRNwl3o09bUq+yDCtZFc1aiwaAAxtcn8YkZXnvk=
github.com/klauspost/cpuid/v2 v2.2.4/go.mod h1:RVVoqg1df56z8g3pUjL/3lE5UfnlrJX8tyFgg4nqhuY=
Expand Down Expand Up @@ -527,6 +531,10 @@ github.com/uptrace/bunrouter v1.0.20 h1:jNvYNcJxF+lSYBQAaQjnE6I11Zs0m+3M5Ek7fq/T
github.com/uptrace/bunrouter v1.0.20/go.mod h1:TwT7Bc0ztF2Z2q/ZzMuSVkcb/Ig/d3MQeP2cxn3e1hI=
github.com/urfave/cli/v2 v2.17.2-0.20221006022127-8f469abc00aa h1:5SqCsI/2Qya2bCzK15ozrqo2sZxkh0FHynJZOTVoV6Q=
github.com/urfave/cli/v2 v2.17.2-0.20221006022127-8f469abc00aa/go.mod h1:1CNUng3PtjQMtRzJO4FMXBQvkGtuYRxxiR9xMa7jMwI=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.51.0 h1:8b30A5JlZ6C7AS81RsWjYMQmrZG6feChmgAolCl1SqA=
github.com/valyala/fasthttp v1.51.0/go.mod h1:oI2XroL+lI7vdXyYoQk03bXBThfFl2cVdIA3Xl7cH8g=
github.com/willf/bitset v1.1.3/go.mod h1:RjeCKbqT1RxIR/KWY6phxZiaY1IyutSBfGjNPySAYV4=
github.com/xlab/treeprint v0.0.0-20180616005107-d6fb6747feb6/go.mod h1:ce1O1j6UtZfjr22oyGxGLbauSBp2YVXpARAosm7dHBg=
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
Expand Down Expand Up @@ -580,8 +588,8 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.16.0 h1:mMMrFzRSCF0GvB7Ne27XVtVAaXLrPmgPC7/v0tkwHaY=
golang.org/x/crypto v0.16.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4=
golang.org/x/crypto v0.17.0 h1:r8bRNjWL3GshPW3gkd+RpvzWrZAwPS49OmTGZ/uhM4k=
golang.org/x/crypto v0.17.0/go.mod h1:gCAAfMLgwOJRpTjQ2zCCt2OcSfYMTeZVSRtQlPC7Nq4=
golang.org/x/exp v0.0.0-20180321215751-8460e604b9de/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20180807140117-3d87b88a115f/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
Expand Down
26 changes: 25 additions & 1 deletion wasmsdk/chunked_upload_progress_storer.go
Original file line number Diff line number Diff line change
@@ -1,14 +1,21 @@
package main

import "github.com/0chain/gosdk/zboxcore/sdk"
import (
"sync"

"github.com/0chain/gosdk/zboxcore/sdk"
)

// chunkedUploadProgressStorer load and save upload progress
type chunkedUploadProgressStorer struct {
list map[string]*sdk.UploadProgress
lock sync.Mutex
}

// Load load upload progress by id
func (mem *chunkedUploadProgressStorer) Load(id string) *sdk.UploadProgress {
mem.lock.Lock()
defer mem.lock.Unlock()
if mem.list == nil {
mem.list = make(map[string]*sdk.UploadProgress)
return nil
Expand All @@ -24,14 +31,31 @@ func (mem *chunkedUploadProgressStorer) Load(id string) *sdk.UploadProgress {

// Save save upload progress
func (mem *chunkedUploadProgressStorer) Save(up sdk.UploadProgress) {
mem.lock.Lock()
defer mem.lock.Unlock()
if mem.list == nil {
mem.list = make(map[string]*sdk.UploadProgress)
}
mem.list[up.ID] = &up
}

//nolint:golint,unused
func (mem *chunkedUploadProgressStorer) Update(id string, chunkIndex int) {
mem.lock.Lock()
defer mem.lock.Unlock()
if mem.list == nil {
return
}
up, ok := mem.list[id]
if ok {
up.ChunkIndex = chunkIndex
}
}

// Remove remove upload progress by id
func (mem *chunkedUploadProgressStorer) Remove(id string) error {
mem.lock.Lock()
defer mem.lock.Unlock()
delete(mem.list, id)
return nil
}
47 changes: 47 additions & 0 deletions zboxcore/mocks/FastClient.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

51 changes: 42 additions & 9 deletions zboxcore/sdk/allocation_file_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import (
"github.com/0chain/gosdk/core/pathutil"
"github.com/0chain/gosdk/core/resty"
"github.com/0chain/gosdk/core/zcncrypto"
"github.com/hitenjain14/fasthttp"

"github.com/0chain/gosdk/zboxcore/blockchain"
zclient "github.com/0chain/gosdk/zboxcore/client"
Expand All @@ -30,7 +31,7 @@ import (
)

func setupHttpResponses(
t *testing.T, mockClient *mocks.HttpClient, allocID string,
t *testing.T, mockClient *mocks.HttpClient, fastMock *mocks.FastClient, allocID string,
refsInput, fileMetaInput []byte, hashes []string,
numBlobbers, numCorrect int, isUpdate bool) {

Expand Down Expand Up @@ -95,6 +96,22 @@ func setupHttpResponses(
return io.NopCloser(bytes.NewReader(b))
}(),
}, nil)
j := i
fastMock.On("DoTimeout", mock.AnythingOfType("*fasthttp.Request"), mock.AnythingOfType("*fasthttp.Response"), mock.AnythingOfType("time.Duration")).Run(func(args mock.Arguments) {
resp := args.Get(1).(*fasthttp.Response)
if j < numCorrect {
resp.Header.SetStatusCode(http.StatusOK)
} else {
resp.Header.SetStatusCode(http.StatusBadRequest)
}
hash := hashes[j]
r := UploadResult{
Filename: "1.txt",
Hash: hash,
}
b, _ := json.Marshal(r)
resp.SetBodyRaw(b)
}).Return(nil)

mockClient.On("Do", mock.MatchedBy(func(req *http.Request) bool {
return req.Method == "POST" &&
Expand Down Expand Up @@ -184,6 +201,8 @@ func setupHttpResponses(
func TestAllocation_UpdateFile(t *testing.T) {
mockClient := mocks.HttpClient{}
zboxutil.Client = &mockClient
mockFastClient := mocks.FastClient{}
zboxutil.FastHttpClient = &mockFastClient

const mockLocalPath = "1.txt"

Expand Down Expand Up @@ -240,7 +259,7 @@ func TestAllocation_UpdateFile(t *testing.T) {
"f435a42af309218e88196d4ed2e0c1977a701641b06434be0bb0263099f3faa9",
"6b3e932bfd2b2c09e39d35e7c4928c42b73bee194045e545560229234d695669",
}
setupHttpResponses(t, &mockClient, a.ID, resfsIn, fileMetaIn, hashes, len(a.Blobbers), len(a.Blobbers), true)
setupHttpResponses(t, &mockClient, &mockFastClient, a.ID, resfsIn, fileMetaIn, hashes, len(a.Blobbers), len(a.Blobbers), true)

err = a.UpdateFile(os.TempDir(), mockLocalPath, "/", nil)
require.NoErrorf(t, err, "Unexpected error %v", err)
Expand All @@ -249,6 +268,8 @@ func TestAllocation_UpdateFile(t *testing.T) {
func TestAllocation_UploadFile(t *testing.T) {
mockClient := mocks.HttpClient{}
zboxutil.Client = &mockClient
mockFastClient := mocks.FastClient{}
zboxutil.FastHttpClient = &mockFastClient

const mockLocalPath = "1.txt"
require := require.New(t)
Expand All @@ -275,7 +296,7 @@ func TestAllocation_UploadFile(t *testing.T) {
"f435a42af309218e88196d4ed2e0c1977a701641b06434be0bb0263099f3faa9",
"6b3e932bfd2b2c09e39d35e7c4928c42b73bee194045e545560229234d695669",
}
setupHttpResponses(t, &mockClient, a.ID, nil, nil, hashes, len(a.Blobbers), len(a.Blobbers), false)
setupHttpResponses(t, &mockClient, &mockFastClient, a.ID, nil, nil, hashes, len(a.Blobbers), len(a.Blobbers), false)

err := a.UploadFile(os.TempDir(), mockLocalPath, "/", nil)
require.NoErrorf(err, "Unexpected error %v", err)
Expand All @@ -289,6 +310,8 @@ func TestAllocation_UpdateFileWithThumbnail(t *testing.T) {

mockClient := mocks.HttpClient{}
zboxutil.Client = &mockClient
mockFastClient := mocks.FastClient{}
zboxutil.FastHttpClient = &mockFastClient

a := &Allocation{
ID: "TestAllocation_UpdateFile_WithThumbNail",
Expand Down Expand Up @@ -345,7 +368,7 @@ func TestAllocation_UpdateFileWithThumbnail(t *testing.T) {
"f435a42af309218e88196d4ed2e0c1977a701641b06434be0bb0263099f3faa9",
"6b3e932bfd2b2c09e39d35e7c4928c42b73bee194045e545560229234d695669",
}
setupHttpResponses(t, &mockClient, a.ID, resfsIn, fileMetaIn, hashes, len(a.Blobbers), len(a.Blobbers), true)
setupHttpResponses(t, &mockClient, &mockFastClient, a.ID, resfsIn, fileMetaIn, hashes, len(a.Blobbers), len(a.Blobbers), true)

err = a.UpdateFileWithThumbnail(os.TempDir(), mockLocalPath, "/", mockThumbnailPath, nil)
require.NoErrorf(t, err, "Unexpected error %v", err)
Expand All @@ -360,6 +383,8 @@ func TestAllocation_UploadFileWithThumbnail(t *testing.T) {

mockClient := mocks.HttpClient{}
zboxutil.Client = &mockClient
mockFastClient := mocks.FastClient{}
zboxutil.FastHttpClient = &mockFastClient

if teardown := setupMockFile(t, mockLocalPath); teardown != nil {
defer teardown(t)
Expand Down Expand Up @@ -388,7 +413,7 @@ func TestAllocation_UploadFileWithThumbnail(t *testing.T) {
"f435a42af309218e88196d4ed2e0c1977a701641b06434be0bb0263099f3faa9",
"6b3e932bfd2b2c09e39d35e7c4928c42b73bee194045e545560229234d695669",
}
setupHttpResponses(t, &mockClient, a.ID, nil, nil, hashes, len(a.Blobbers), len(a.Blobbers), false)
setupHttpResponses(t, &mockClient, &mockFastClient, a.ID, nil, nil, hashes, len(a.Blobbers), len(a.Blobbers), false)

err := a.UploadFileWithThumbnail(mockTmpPath, mockLocalPath, "/", mockThumbnailPath, nil)
require.NoErrorf(t, err, "Unexpected error %v", err)
Expand All @@ -397,6 +422,8 @@ func TestAllocation_UploadFileWithThumbnail(t *testing.T) {
func TestAllocation_EncryptAndUpdateFile(t *testing.T) {
mockClient := mocks.HttpClient{}
zboxutil.Client = &mockClient
mockFastClient := mocks.FastClient{}
zboxutil.FastHttpClient = &mockFastClient

const mockLocalPath = "1.txt"

Expand Down Expand Up @@ -452,7 +479,7 @@ func TestAllocation_EncryptAndUpdateFile(t *testing.T) {
"3c4f6a43748f6b7cefee11216540414cb9b2563c294a5f7d633c2e9cda26f7bc",
"249684daaeef1a8d38d0be0ea38777886e0b3ddf3deaef2eabe4117cc6e67256",
}
setupHttpResponses(t, &mockClient, a.ID, resfsIn, fileMetaIn, hashes, len(a.Blobbers), len(a.Blobbers), true)
setupHttpResponses(t, &mockClient, &mockFastClient, a.ID, resfsIn, fileMetaIn, hashes, len(a.Blobbers), len(a.Blobbers), true)

err = a.EncryptAndUpdateFile(os.TempDir(), mockLocalPath, "/", nil)
require.NoError(t, err)
Expand All @@ -461,6 +488,8 @@ func TestAllocation_EncryptAndUpdateFile(t *testing.T) {
func TestAllocation_EncryptAndUploadFile(t *testing.T) {
mockClient := mocks.HttpClient{}
zboxutil.Client = &mockClient
mockFastClient := mocks.FastClient{}
zboxutil.FastHttpClient = &mockFastClient

const (
mockLocalPath = "1.txt"
Expand Down Expand Up @@ -491,7 +520,7 @@ func TestAllocation_EncryptAndUploadFile(t *testing.T) {
"f435a42af309218e88196d4ed2e0c1977a701641b06434be0bb0263099f3faa9",
"6b3e932bfd2b2c09e39d35e7c4928c42b73bee194045e545560229234d695669",
}
setupHttpResponses(t, &mockClient, a.ID, nil, nil, hashes, len(a.Blobbers), len(a.Blobbers), false)
setupHttpResponses(t, &mockClient, &mockFastClient, a.ID, nil, nil, hashes, len(a.Blobbers), len(a.Blobbers), false)

err := a.EncryptAndUploadFile(mockTmpPath, mockLocalPath, "/", nil)
require.NoError(t, err)
Expand All @@ -500,6 +529,8 @@ func TestAllocation_EncryptAndUploadFile(t *testing.T) {
func TestAllocation_EncryptAndUpdateFileWithThumbnail(t *testing.T) {
mockClient := mocks.HttpClient{}
zboxutil.Client = &mockClient
mockFastClient := mocks.FastClient{}
zboxutil.FastHttpClient = &mockFastClient

const (
mockLocalPath = "1.txt"
Expand Down Expand Up @@ -565,7 +596,7 @@ func TestAllocation_EncryptAndUpdateFileWithThumbnail(t *testing.T) {
"3c4f6a43748f6b7cefee11216540414cb9b2563c294a5f7d633c2e9cda26f7bc",
"249684daaeef1a8d38d0be0ea38777886e0b3ddf3deaef2eabe4117cc6e67256",
}
setupHttpResponses(t, &mockClient, a.ID, resfsIn, fileMetaIn, hashes, len(a.Blobbers), len(a.Blobbers), true)
setupHttpResponses(t, &mockClient, &mockFastClient, a.ID, resfsIn, fileMetaIn, hashes, len(a.Blobbers), len(a.Blobbers), true)
err = a.EncryptAndUpdateFileWithThumbnail(mockTmpPath, mockLocalPath, "/", mockThumbnailPath, nil)

require.NoError(t, err)
Expand All @@ -574,6 +605,8 @@ func TestAllocation_EncryptAndUpdateFileWithThumbnail(t *testing.T) {
func TestAllocation_EncryptAndUploadFileWithThumbnail(t *testing.T) {
mockClient := mocks.HttpClient{}
zboxutil.Client = &mockClient
mockFastClient := mocks.FastClient{}
zboxutil.FastHttpClient = &mockFastClient

const (
mockLocalPath = "1.txt"
Expand Down Expand Up @@ -612,7 +645,7 @@ func TestAllocation_EncryptAndUploadFileWithThumbnail(t *testing.T) {
"6b3e932bfd2b2c09e39d35e7c4928c42b73bee194045e545560229234d695669",
}

setupHttpResponses(t, &mockClient, a.ID, nil, nil, hashes, len(a.Blobbers), len(a.Blobbers), false)
setupHttpResponses(t, &mockClient, &mockFastClient, a.ID, nil, nil, hashes, len(a.Blobbers), len(a.Blobbers), false)

err := a.EncryptAndUploadFileWithThumbnail(mockTmpPath, mockLocalPath, "/", mockThumbnailPath, nil)
require.NoError(t, err)
Expand Down
Loading

0 comments on commit acaf667

Please sign in to comment.