Skip to content
This repository has been archived by the owner on Aug 8, 2024. It is now read-only.

Jkl 330 implement the new file system #140

Merged
merged 94 commits into from
Apr 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
da826a4
add WriteToDisk which saves data as single physical file instead of b…
dahn510 Jul 12, 2023
256aa32
simplify and reduce responsibility of CreateMerkleTree() and WriteToD…
dahn510 Jul 13, 2023
06b6046
add bytes written return value to WriteToDisk()
dahn510 Jul 13, 2023
ce70e86
update saveFile()
dahn510 Jul 13, 2023
dfeac44
createFID and CreateMerkleTree() now uses io.reader and io.seeker to …
dahn510 Jul 14, 2023
1c35348
add GetBlock()
dahn510 Jul 14, 2023
10ab863
remove blockSize from FileServer for now
dahn510 Jul 17, 2023
b4d758d
add GetStorageDirForTree() and GetFileNameForTree()
dahn510 Jul 17, 2023
1d36ce5
migrate fileserver to use new method of saving files to disk
dahn510 Jul 17, 2023
ac89cf6
undo GetStoragePath() ctx change to utils.Context
dahn510 Jul 18, 2023
df4c953
gofumpt
dahn510 Jul 18, 2023
14c369a
break down GetStoragePath()
dahn510 Jul 18, 2023
8927b31
initial draft for migrating old fs to new fs
dahn510 Jul 18, 2023
2c21265
second draft for migrating old fs to new fs
dahn510 Jul 18, 2023
9b36e69
add GetPiece() and blockSize to FileServer
dahn510 Jul 20, 2023
ecbf46a
utils: add postGlueCheck() and DiscoverFids() for migration
dahn510 Jul 21, 2023
f5b4f1a
add comments to functions
dahn510 Jul 21, 2023
7a6c8f7
add Migrate command to root
dahn510 Jul 22, 2023
b6c0842
remove garbage
dahn510 Jul 22, 2023
deef39e
hook new fs to postProof
dahn510 Jul 25, 2023
a99e1aa
fixing bugs
dahn510 Jul 27, 2023
56b8c4a
fix test error
dahn510 Jul 28, 2023
25fa0f5
fix file server and migrate tool
dahn510 Jul 28, 2023
699c7cd
fix broken hands
dahn510 Jul 28, 2023
4fa9221
fix migration cmd
dahn510 Jul 28, 2023
eecc6c2
lint clean up
dahn510 Jul 28, 2023
b249b57
change how test files are written on disk
dahn510 Aug 1, 2023
cedef2c
add a test case to TestGetPiece that test getting a block starting al…
dahn510 Aug 1, 2023
07fbbd0
adding the test case
dahn510 Aug 1, 2023
3b85935
moving stray service initialization to start command function
dahn510 Aug 9, 2023
3016538
add archive module
dahn510 Aug 18, 2023
78650a1
add tree management to archive interface
dahn510 Aug 18, 2023
a39095c
intergrating archive module to existing system
dahn510 Sep 4, 2023
9c4c78c
add archivedb
dahn510 Sep 8, 2023
aa34361
ArchiveDB implementation
dahn510 Sep 11, 2023
7bc7e27
downtime db
dahn510 Sep 17, 2023
833609d
wire new db modules
dahn510 Sep 18, 2023
64f0319
fix wiring
dahn510 Sep 19, 2023
970df96
fix merkle tree file dir generation
dahn510 Sep 19, 2023
83550b3
fix stray manager nil reference to archive
dahn510 Sep 19, 2023
e39730c
fix EOF bug in LittleHand Process() and faulty check if file exists
dahn510 Sep 20, 2023
f843237
fix proof building in GenerateProof
dahn510 Oct 3, 2023
bf30a8a
test CreateMerkleTree
dahn510 Oct 4, 2023
7db95cf
fix archive GetPiece unit test
dahn510 Oct 4, 2023
0f174ad
contract db get fid test
dahn510 Oct 23, 2023
ab1f6fa
GetContracts unit test
dahn510 Oct 24, 2023
50bb8e0
archive SetContract unit test
dahn510 Oct 25, 2023
7a8068f
test archive SetContract reverse reference
dahn510 Oct 25, 2023
0f6003a
fix addReference
dahn510 Oct 25, 2023
660c44f
unit test archive delete contract
dahn510 Oct 31, 2023
46057ea
fix proof block index bug
dahn510 Oct 31, 2023
cbbd882
query to check cid is mine
dahn510 Nov 2, 2023
e42b2ca
split contract clean up with posting proofs
dahn510 Nov 8, 2023
3c8f75a
split contract clean up with posting proofs
dahn510 Nov 8, 2023
14a19c8
Merge branch 'JKL-330-implement-the-new-file-system' of github.com:Ja…
dahn510 Nov 8, 2023
c187722
Merge branch 'JKL-330-implement-the-new-file-system' of github.com:Ja…
dahn510 Nov 15, 2023
bfbf262
update postProofs service to StartProofServer
dahn510 Nov 22, 2023
3ceeb8c
update TestGenerateMerkleProof and add benchmark
dahn510 Nov 26, 2023
f1c4606
Remove deprecated tests
dahn510 Nov 26, 2023
a5c898a
bug fix for post proofs
dahn510 Dec 11, 2023
2e6a94c
break down littleHand Process
dahn510 Dec 26, 2023
50121e7
hex encode merkle root
dahn510 Dec 30, 2023
15495bb
fix data not found on post proof
dahn510 Jan 20, 2024
07a40e6
remove commented codes
dahn510 Jan 20, 2024
c0a8488
migrate also move merkle tree file to new location
dahn510 Jan 25, 2024
f068756
clean old blocks
dahn510 Jan 25, 2024
a69f681
upgrade canine-chain module to canine-chain/v3
dahn510 Feb 20, 2024
c735810
Revert "upgrade canine-chain module to canine-chain/v3"
dahn510 Feb 21, 2024
8437788
remove dumb comments
dahn510 Feb 21, 2024
36ff819
Improve migration process
dahn510 Feb 27, 2024
25511d2
fix migration bug and more information during migration
dahn510 Feb 28, 2024
bbd28e6
update packages and bump makefile go version check to 1.22
dahn510 Mar 12, 2024
f2a3842
test migration script tweak
dahn510 Mar 12, 2024
ce85f77
fix proof failure
dahn510 Mar 13, 2024
ea791cd
Merge branch 'main' of github.com:JackalLabs/canine-provider into JKL…
dahn510 Mar 18, 2024
d7dea0b
gofumpt
dahn510 Mar 18, 2024
20ed154
replace old canine-chain package to v3
dahn510 Mar 25, 2024
354a04c
rework attest error response
dahn510 Mar 27, 2024
bfa6e32
lint warnings fix and code formatting
dahn510 Apr 2, 2024
672f509
use chunksize cmd flag for migrate cmd
dahn510 Apr 2, 2024
846f875
gofumpt
dahn510 Apr 2, 2024
639c210
Gracefully shutdown http server and databases
dahn510 Apr 5, 2024
9e7635c
recollects contracts missing in db on startup
dahn510 Apr 7, 2024
fad726b
fix little hand initialization process
dahn510 Apr 9, 2024
0cfc51b
filter strays that aren't already claimed stray
dahn510 Apr 9, 2024
f4d6a75
fix filtering claimed strays
dahn510 Apr 9, 2024
94c4e32
display error during downfil
dahn510 Apr 12, 2024
7bb0d19
Merge branch 'main' of github.com:JackalLabs/canine-provider into JKL…
dahn510 Apr 15, 2024
45096fc
gofumpt
dahn510 Apr 15, 2024
a1a54b9
update go versiom in actions & lint
TheMarstonConnell Apr 15, 2024
2c28e56
Merge pull request #141 from JackalLabs/marston/lint-go-version
TheMarstonConnell Apr 15, 2024
466835e
fix unit test expected output
dahn510 Apr 15, 2024
02ae07f
fix TestGetPiece expected output
dahn510 Apr 15, 2024
4ba0263
lint
TheMarstonConnell Apr 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,6 @@ jobs:
- uses: actions/[email protected]
- uses: actions/setup-go@v5
with:
go-version: 1.19 # The Go version to download (if necessary) and use.
go-version: 1.22 # The Go version to download (if necessary) and use.
- name: Build CLI
run: go install ./...
2 changes: 1 addition & 1 deletion .github/workflows/cov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
- uses: actions/[email protected]
- uses: actions/setup-go@v5
with:
go-version: 1.19 # The Go version to download (if necessary) and use.
go-version: 1.22 # The Go version to download (if necessary) and use.
- name: Get Coverage
run: |
go test ./... -cover -coverprofile coverage.out
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
steps:
- uses: actions/setup-go@v5
with:
go-version: 1.19
go-version: 1.22
- uses: actions/[email protected]
- name: golangci-lint
uses: golangci/golangci-lint-action@v4
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- uses: actions/[email protected]
- uses: actions/setup-go@v5
with:
go-version: 1.18.1 # The Go version to download (if necessary) and use.
go-version: 1.22 # The Go version to download (if necessary) and use.
- name: Build CLI
shell: bash
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- uses: actions/[email protected]
- uses: actions/setup-go@v5
with:
go-version: 1.18.1 # The Go version to download (if necessary) and use.
go-version: 1.22 # The Go version to download (if necessary) and use.
- name: Build CLI
shell: bash
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- uses: actions/[email protected]
- uses: actions/setup-go@v5
with:
go-version: 1.19 # The Go version to download (if necessary) and use.
go-version: 1.22 # The Go version to download (if necessary) and use.
- name: Test
shell: bash
run: go test ./...
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ benchmark:
###############################################################################

format-tools:
go install mvdan.cc/gofumpt@v0.3.1
go install mvdan.cc/gofumpt@v0.6.0
gofumpt -l -w .

lint: format-tools
Expand All @@ -139,7 +139,7 @@ format: format-tools

# Add check to make sure we are using the proper Go version before proceeding with anything
check-go-version:
@if ! go version | grep -q "go1.20"; then \
@if ! go version | grep -q "go1.22"; then \
echo "\033[0;31mERROR:\033[0m Go version 1.20 is required for compiling canined. It looks like you are using" "$(shell go version) \nThere are potential consensus-breaking changes that can occur when running binaries compiled with different versions of Go. Please download Go version 1.20 and retry. Thank you!"; \
exit 1; \
fi
2 changes: 1 addition & 1 deletion benchmark/bench.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ import (

"github.com/wealdtech/go-merkletree/sha3"

"github.com/wealdtech/go-merkletree"
merkletree "github.com/wealdtech/go-merkletree"
)

func main() {
Expand Down
41 changes: 21 additions & 20 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/JackalLabs/jackal-provider

go 1.20
go 1.22.0

require (
github.com/JackalLabs/blanket v0.0.0
Expand All @@ -9,20 +9,21 @@ require (
github.com/huin/goupnp v1.3.0
github.com/jackalLabs/canine-chain/v3 v3.0.0
github.com/julienschmidt/httprouter v1.3.0
github.com/rs/cors v1.10.1
github.com/rs/zerolog v1.32.0
github.com/rs/cors v1.9.0
github.com/rs/zerolog v1.29.1
github.com/spf13/cobra v1.8.0
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.16.0
github.com/stretchr/testify v1.8.4
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7
github.com/tendermint/tendermint v0.34.27
github.com/wealdtech/go-merkletree v1.0.0
golang.org/x/sync v0.7.0
golang.org/x/sync v0.6.0
google.golang.org/grpc v1.57.0
)

require (
cosmossdk.io/api v0.2.6 // indirect
cosmossdk.io/api v0.4.0 // indirect
cosmossdk.io/core v0.5.1 // indirect
cosmossdk.io/depinject v1.0.0-alpha.3 // indirect
filippo.io/edwards25519 v1.0.0-rc.1 // indirect
Expand Down Expand Up @@ -50,7 +51,7 @@ require (
github.com/cosmos/btcutil v1.0.4 // indirect
github.com/cosmos/cosmos-db v0.0.0-20221226095112-f3c38ecb5e32 // indirect
github.com/cosmos/cosmos-proto v1.0.0-beta.3 // indirect
github.com/cosmos/gogoproto v1.4.10 // indirect
github.com/cosmos/gogoproto v1.4.11 // indirect
github.com/cosmos/gorocksdb v1.2.0 // indirect
github.com/cosmos/iavl v0.19.5 // indirect
github.com/cosmos/ibc-go/v4 v4.6.0 // indirect
Expand Down Expand Up @@ -82,10 +83,10 @@ require (
github.com/gogo/gateway v1.1.0 // indirect
github.com/gogo/protobuf v1.3.3 // indirect
github.com/golang/glog v1.0.0 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/golang/snappy v0.0.5-0.20220116011046-fa5810519dcb // indirect
github.com/google/btree v1.1.2 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/orderedcode v0.0.1 // indirect
github.com/google/uuid v1.3.0 // indirect
Expand All @@ -98,7 +99,7 @@ require (
github.com/gtank/merlin v0.1.1 // indirect
github.com/gtank/ristretto255 v0.1.2 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d // indirect
github.com/hashicorp/golang-lru v1.0.2 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/hdevalence/ed25519consensus v0.0.0-20220222234857-c00d1f31bab3 // indirect
github.com/improbable-eng/grpc-web v0.15.0 // indirect
Expand All @@ -115,8 +116,8 @@ require (
github.com/magiconair/properties v1.8.7 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mattn/go-runewidth v0.0.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
github.com/mimoo/StrobeGo v0.0.0-20210601165009-122bf33a46e0 // indirect
github.com/minio/highwayhash v1.0.2 // indirect
Expand All @@ -135,9 +136,10 @@ require (
github.com/rakyll/statik v0.1.7 // indirect
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475 // indirect
github.com/regen-network/cosmos-proto v0.3.1 // indirect
github.com/rivo/uniseg v0.2.0 // indirect
github.com/rogpeppe/go-internal v1.9.0 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/rogpeppe/go-internal v1.11.0 // indirect
github.com/sasha-s/go-deadlock v0.3.1 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/spf13/afero v1.9.5 // indirect
github.com/spf13/cast v1.5.1 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
Expand All @@ -149,15 +151,14 @@ require (
github.com/zondax/hid v0.9.1 // indirect
github.com/zondax/ledger-go v0.14.1 // indirect
go.etcd.io/bbolt v1.3.7 // indirect
golang.org/x/crypto v0.17.0 // indirect
golang.org/x/exp v0.0.0-20230515195305-f3d0a9c9a5cc // indirect
golang.org/x/net v0.17.0 // indirect
golang.org/x/sys v0.15.0 // indirect
golang.org/x/term v0.15.0 // indirect
golang.org/x/crypto v0.12.0 // indirect
golang.org/x/exp v0.0.0-20240222234643-814bf88cf225 // indirect
golang.org/x/net v0.14.0 // indirect
golang.org/x/sys v0.18.0 // indirect
golang.org/x/term v0.11.0 // indirect
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect
google.golang.org/grpc v1.55.0 // indirect
google.golang.org/protobuf v1.30.0 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
Expand Down
Loading
Loading