Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: full blocks, bundling & compression, blob usage optimization & tracking #125

Draft
wants to merge 178 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
178 commits
Select commit Hold shift + click to select a range
994aaee
use alloy rs
MujkicA Aug 20, 2024
4379088
fix aws config
MujkicA Aug 21, 2024
c774288
remove ethers
MujkicA Aug 26, 2024
b8c9ae0
remove debug
MujkicA Aug 26, 2024
43a93df
cleanup
MujkicA Aug 26, 2024
a43231b
aws region to aws config
MujkicA Aug 26, 2024
df9bfd8
remove debug items
MujkicA Aug 26, 2024
acf85b2
remove unwrap
MujkicA Aug 26, 2024
a74458f
add logs
MujkicA Aug 26, 2024
7e6f65b
dbg log err msg
MujkicA Aug 26, 2024
d6272e3
fmt
MujkicA Aug 26, 2024
524fe22
set region on credentials provider
MujkicA Aug 26, 2024
edcb82a
add logs
MujkicA Aug 26, 2024
8f43edc
dbg log err msg
MujkicA Aug 26, 2024
fbd0dd1
fmt
MujkicA Aug 26, 2024
4d3c9da
Merge branch 'feat/blob-logging' of github.com:FuelLabs/fuel-block-co…
MujkicA Aug 26, 2024
832736b
add e2e tests
MujkicA Aug 27, 2024
be86b92
rename var
MujkicA Aug 27, 2024
35e3bf9
add wait for blob
MujkicA Aug 28, 2024
fc1430b
add blob e2e test
MujkicA Aug 28, 2024
dabc4e0
enable block validation
MujkicA Aug 28, 2024
3a7ba67
fix blob tx
MujkicA Aug 28, 2024
138bda4
remove nonce setting
MujkicA Aug 28, 2024
ea1b17b
Merge remote-tracking branch 'origin/master' into feat/alloy-rs
MujkicA Aug 29, 2024
78fc9f7
use separate provider for blobs
MujkicA Aug 29, 2024
69d8d8f
Merge branch 'feat/alloy-rs' into feat/blob-logging
MujkicA Aug 29, 2024
d61237c
improvements
MujkicA Aug 29, 2024
08ca4e9
rename id to arn
MujkicA Aug 29, 2024
06a43a1
change envs to arn
MujkicA Aug 29, 2024
2816f7a
improve logs
MujkicA Aug 29, 2024
fbc61ea
pr comments
hal3e Aug 29, 2024
036754f
import behind flag and do not log in tests
hal3e Aug 29, 2024
9e944e3
merge with base
MujkicA Aug 29, 2024
1569d38
Merge remote-tracking branch 'origin/master' into feat/blob-logging
MujkicA Aug 29, 2024
c356cb0
fix for concrurent e2e tests
segfault-magnet Aug 29, 2024
6923e88
fix env variable
segfault-magnet Aug 30, 2024
5923cfc
clippy
segfault-magnet Aug 30, 2024
1f9251a
group imports, reorder
segfault-magnet Aug 30, 2024
7f4a03b
cargo sort
segfault-magnet Aug 30, 2024
8bb7728
Merge remote-tracking branch 'origin/master' into feat/blob-logging
segfault-magnet Aug 30, 2024
fb978b8
cleanup, fix cargo deps
segfault-magnet Aug 30, 2024
34c017a
snapshot dir must be kept alive
segfault-magnet Aug 30, 2024
5e7be0c
add finalized_at to l1 transactions indicating when the tx was finalized
segfault-magnet Sep 3, 2024
f769949
add clock port and a adapter
segfault-magnet Sep 3, 2024
8efff99
will wait for data accumulation until timeout triggers
segfault-magnet Sep 3, 2024
db3c402
fixing tests
segfault-magnet Sep 3, 2024
fe946c0
Merge remote-tracking branch 'origin/master' into feat/blob_fragmenta…
segfault-magnet Sep 3, 2024
aede9ac
add fallback for last finalized time
segfault-magnet Sep 3, 2024
247a2a6
wip
segfault-magnet Sep 3, 2024
2478611
syncing
segfault-magnet Sep 5, 2024
080df5e
migration fixed
segfault-magnet Sep 5, 2024
05424a7
testing out the query
segfault-magnet Sep 5, 2024
604a8c2
wip
segfault-magnet Sep 6, 2024
cdfba9f
wip
segfault-magnet Sep 9, 2024
b063dec
wip
segfault-magnet Sep 9, 2024
38b3265
wip
segfault-magnet Sep 9, 2024
789c96b
importer first version finished
segfault-magnet Sep 9, 2024
67ccaeb
wip
segfault-magnet Sep 10, 2024
5542b91
wip
segfault-magnet Sep 11, 2024
0983d0a
fix tests
segfault-magnet Sep 11, 2024
9e08da4
guarding against empty blocks and fragments
segfault-magnet Sep 11, 2024
fd1c495
wip
segfault-magnet Sep 11, 2024
d4b6403
fixing non zero vec changes
segfault-magnet Sep 11, 2024
cf3dac4
finished changes for nonemptyvec
segfault-magnet Sep 11, 2024
e1f06c7
cleanup
segfault-magnet Sep 12, 2024
9ed485f
remove test only db code
segfault-magnet Sep 13, 2024
3f53a76
refactor tests
segfault-magnet Sep 13, 2024
2bff202
setup structure for generating services
segfault-magnet Sep 14, 2024
8385bcf
decoupled tests from implementation
segfault-magnet Sep 14, 2024
dd84672
move test utils
segfault-magnet Sep 14, 2024
2a70ce9
cleanup
segfault-magnet Sep 14, 2024
d73a70e
prepare for price optimization
segfault-magnet Sep 14, 2024
00ab746
fix tests
segfault-magnet Sep 14, 2024
f1978e5
factory for block bundle proposers
segfault-magnet Sep 14, 2024
0b3bfe6
test for stopping of optimization runs
segfault-magnet Sep 14, 2024
fe383bd
test for incremental optimization
segfault-magnet Sep 14, 2024
b06e4cb
cleanup
segfault-magnet Sep 14, 2024
5ae5537
tests passing, cleanup
segfault-magnet Sep 14, 2024
e56c513
cleanup
segfault-magnet Sep 14, 2024
79248cb
split into files
segfault-magnet Sep 14, 2024
a166c3c
cleanup
segfault-magnet Sep 14, 2024
b0304b2
helper for encoding blocks
segfault-magnet Sep 15, 2024
dbb8fcd
cleaning up importer tests
segfault-magnet Sep 15, 2024
059931d
cleanup of importer tests finished
segfault-magnet Sep 15, 2024
f14c723
refactor state committer
segfault-magnet Sep 15, 2024
07d10d4
add tests
segfault-magnet Sep 15, 2024
51f8fbe
move to block importer, add tests
segfault-magnet Sep 15, 2024
0738d0b
gas per byte
segfault-magnet Sep 15, 2024
f7ab599
compression of the bundle
segfault-magnet Sep 15, 2024
4dbe8a5
refactoring bundler
segfault-magnet Sep 16, 2024
6373371
tests passing
segfault-magnet Sep 16, 2024
045f1fa
cleaning up
segfault-magnet Sep 16, 2024
2d4bcf3
non blocking compression
segfault-magnet Sep 16, 2024
25a28c0
refactorings
segfault-magnet Sep 16, 2024
d9ddcb3
add last_time_a_fragment_was_finalized
segfault-magnet Sep 16, 2024
b169580
tests passing
segfault-magnet Sep 16, 2024
02fc314
finished test for finalization timeout
segfault-magnet Sep 16, 2024
4c1bb6a
moving towards separating bundling from gas estimating
segfault-magnet Sep 16, 2024
815ead3
tests fixed
segfault-magnet Sep 17, 2024
ffcd87a
gas calculations finished
segfault-magnet Sep 17, 2024
0380040
e2e test ran, problems with genesis block validation
segfault-magnet Sep 17, 2024
ab68823
fix e2e
segfault-magnet Sep 18, 2024
cd11c60
tests passing
segfault-magnet Sep 18, 2024
1f743d5
dry up tests, remove unnecessary result
segfault-magnet Sep 18, 2024
ddcc4ad
guard against nonsequential blocks
segfault-magnet Sep 18, 2024
b50c566
log improvement
segfault-magnet Sep 18, 2024
115f9d8
sqlx prepare
segfault-magnet Sep 19, 2024
ea0b597
testing the bundler
segfault-magnet Sep 19, 2024
dfed727
moved gas calculations away from the websocket connection
segfault-magnet Sep 19, 2024
0d25cc6
dry up block encoding
segfault-magnet Sep 19, 2024
e8470e4
added tests to bundler
segfault-magnet Sep 19, 2024
8ef6e39
finished bundler tests
segfault-magnet Sep 19, 2024
2e2dcbc
opt for starting height instead of lookback window for more predictable
segfault-magnet Sep 20, 2024
1b0d6db
config updated
segfault-magnet Sep 20, 2024
f3e526d
remove config file, update e2e tests with termination condition
segfault-magnet Sep 20, 2024
cf82879
enable all tests, sqlx prepare
segfault-magnet Sep 20, 2024
a6b9f38
use blocks endpoint with windowed ranges
hal3e Sep 20, 2024
6907e9b
remove warnings
hal3e Sep 20, 2024
0a7336b
add buil.rs for fuel package and add client_ext code
hal3e Sep 20, 2024
a234421
remove approx
segfault-magnet Sep 20, 2024
6a28f98
storage uses plain async trait methods, DbWithProcess is now a thing
segfault-magnet Sep 20, 2024
e3018d5
move block_ext to client folder
hal3e Sep 20, 2024
567697d
delegate macro and removal of async trait
segfault-magnet Sep 20, 2024
ccfc3f3
Merge branch 'feat/blob_fragmentation' of github.com:FuelLabs/fuel-bl…
segfault-magnet Sep 20, 2024
8476084
batch inserting blocks, tests for db
segfault-magnet Sep 20, 2024
cd551cd
full blocks now imported
segfault-magnet Sep 21, 2024
ba76c67
tweak e2e test and config
segfault-magnet Sep 21, 2024
7f72494
cleaning up
segfault-magnet Sep 21, 2024
b466a44
decouple validating from validated block
segfault-magnet Sep 21, 2024
c6b24d3
cleanup
segfault-magnet Sep 21, 2024
fb12dd8
new api for fragmenting
segfault-magnet Sep 22, 2024
e6a8fde
e2e tests passing with blobs as fragments
segfault-magnet Sep 22, 2024
01d4c8f
checkpoint before extracing bundling into a separate service
segfault-magnet Sep 22, 2024
8144010
separation into two services
segfault-magnet Sep 22, 2024
de1e04b
e2e test passing
segfault-magnet Sep 22, 2024
11208e3
can connect to testnet
segfault-magnet Sep 22, 2024
5fcde56
add blob utilization metrics
segfault-magnet Sep 23, 2024
620ac4c
cleanup
segfault-magnet Sep 23, 2024
3f378bf
lower limit to speedup e2e test
segfault-magnet Sep 23, 2024
1b98a94
swap out local nonempty impl for crate
segfault-magnet Sep 23, 2024
72fe59c
cleanup
segfault-magnet Sep 23, 2024
b3abcf0
group imports
segfault-magnet Sep 23, 2024
22b1913
remove unwraps
segfault-magnet Sep 23, 2024
b0eb64e
added tests for SequentialFuelBlocks
segfault-magnet Sep 23, 2024
aecb81b
configurable amount of full blocks per request
segfault-magnet Sep 23, 2024
1067748
batch inserting fragments
segfault-magnet Sep 23, 2024
79a7002
fix stackoverflow
segfault-magnet Sep 23, 2024
55acc67
cargo sort
segfault-magnet Sep 23, 2024
4265276
sqlx json update
segfault-magnet Sep 23, 2024
283c64c
remove unused deps
segfault-magnet Sep 23, 2024
2083abc
rename leftover state committer vars to block bundlers
segfault-magnet Sep 23, 2024
5da7972
remove print statements
segfault-magnet Sep 23, 2024
75c54a3
added fuel api to bundler
segfault-magnet Sep 23, 2024
70cd719
added current height check to block bundler
segfault-magnet Sep 23, 2024
68e23e1
block importer now actively tracks lookback window
segfault-magnet Sep 23, 2024
6fb2e4c
state committer now tracks lookback window
segfault-magnet Sep 23, 2024
09abe8c
add test for lookback window
segfault-magnet Sep 23, 2024
7e4a842
fix metrics add comments
segfault-magnet Sep 23, 2024
1c353ca
update sqlx
segfault-magnet Sep 23, 2024
a11795e
cleanup
segfault-magnet Sep 23, 2024
d453f36
fix remaining tests
segfault-magnet Sep 23, 2024
01ebc40
fix migration, add test for migration script
segfault-magnet Sep 24, 2024
7133985
format
segfault-magnet Sep 24, 2024
e249719
optimization step
segfault-magnet Sep 24, 2024
7a79164
e2e run script
segfault-magnet Sep 24, 2024
5be62d2
parallellize bundle optimization
segfault-magnet Sep 24, 2024
770a7fb
add config for blobs per optimization run
segfault-magnet Sep 24, 2024
62d74c1
amount of pending tx query
segfault-magnet Sep 24, 2024
40480d5
added clock to state committer
segfault-magnet Sep 24, 2024
75e110d
fragments to accumulate config
segfault-magnet Sep 24, 2024
c7ba620
Revert "amount of pending tx query"
segfault-magnet Sep 24, 2024
b6678e7
reenable live sqlx
segfault-magnet Sep 24, 2024
6c0aca8
added fragment accumulation
segfault-magnet Sep 24, 2024
c2de0d8
tests passing
segfault-magnet Sep 24, 2024
9834124
adapt e2e
segfault-magnet Sep 24, 2024
d3ae323
restore tests
segfault-magnet Sep 24, 2024
06717a7
fmt
segfault-magnet Sep 24, 2024
1dbdcc9
add some config validation
segfault-magnet Sep 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Generated by Cargo
# will have compiled files and executables
/target/
**/target/

# These are backup files generated by rustfmt
**/*.rs.bk
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

This file was deleted.

This file was deleted.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

This file was deleted.

Loading
Loading