Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add bare Chain Replication implementation & a bunch of fixes #20

Merged
merged 328 commits into from
Apr 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
328 commits
Select commit Hold shift + click to select a range
f30dd2b
fixed snapshotting commit_bar bug
josehu07 Oct 4, 2023
8dd10a2
add chunking to reconstruction read messages
josehu07 Oct 4, 2023
8751839
update counstraint boundary figure
josehu07 Oct 5, 2023
e7a4e96
make follower gossiping optimal
josehu07 Oct 5, 2023
ca6f52e
staging progress on peer health tracking
josehu07 Oct 6, 2023
5b1c107
finish peer health tracking
josehu07 Oct 6, 2023
313dba1
finish peer health tracking
josehu07 Oct 6, 2023
1e8a8a0
add fallback mechanism to Crossword
josehu07 Oct 6, 2023
d0d14ef
Merge pull request #7 from josehu07/crossword
josehu07 Oct 6, 2023
1b04307
minor updates to README
josehu07 Oct 6, 2023
68066d9
start working on Raft impl
josehu07 Oct 6, 2023
90b13fd
rename LogEntry to WalEntry for Paxos-style impls
josehu07 Oct 7, 2023
7d69552
basically finished Raft impl
josehu07 Oct 8, 2023
63591b6
finish raft impl & debugging
josehu07 Oct 8, 2023
5046591
make Paxos variants snapshotting conservative
josehu07 Oct 8, 2023
913dfa6
add missing hole filling mechanism to Paxos variants
josehu07 Oct 8, 2023
8fd45a3
add Raft to workflow proc tests
josehu07 Oct 9, 2023
cb7f738
Merge pull request #8 from josehu07/raft
josehu07 Oct 9, 2023
ffbe9b4
minor updates to README
josehu07 Oct 9, 2023
cfcd0f4
staging progress on CRaft impl
josehu07 Oct 10, 2023
e11fb98
checked proper logging for critical actions
josehu07 Oct 10, 2023
eab87bf
update local_cluster.py script & bug fixes
josehu07 Oct 10, 2023
d566faf
almost finished CRaft impl
josehu07 Oct 10, 2023
bd8300d
almost finished CRaft impl
josehu07 Oct 10, 2023
ef77bd9
finished CRaft impl
josehu07 Oct 10, 2023
ce12cd9
Merge pull request #9 from josehu07/crossword
josehu07 Oct 10, 2023
273e68d
update constraint boundary figure
josehu07 Oct 11, 2023
0901d89
add probabilistic modeling script
josehu07 Oct 12, 2023
1a568de
staging progress on prob calculation
josehu07 Oct 13, 2023
33d9664
kinda finish distribution model calculation
josehu07 Oct 13, 2023
997835a
make unnessary asserts debug_asserts
josehu07 Oct 13, 2023
10f9763
improving Raft's async performance
josehu07 Oct 13, 2023
2aca29c
add next_slot decrement bypassing
josehu07 Oct 13, 2023
b6531c5
add next_slot decrement bypassing
josehu07 Oct 13, 2023
bfdb361
Merge pull request #10 from josehu07/crossword
josehu07 Oct 13, 2023
d6edaae
relocate workflow_test.py
josehu07 Oct 14, 2023
95929d5
update prob modeling script
josehu07 Oct 14, 2023
4b14eee
update constraint boundary figure script
josehu07 Oct 14, 2023
52f199a
add pin CPU cores feature to scripts
josehu07 Oct 16, 2023
3507bea
staging progress on eval scripts
josehu07 Oct 16, 2023
25dc5e2
staging progress on automating bench scripts
josehu07 Oct 17, 2023
e7d32e3
staging progress on automating bench scripts
josehu07 Oct 17, 2023
603ee81
fixed automated bench script issues
josehu07 Oct 17, 2023
10f3118
minor updates to prob calculation script
josehu07 Oct 18, 2023
edea398
update modeling script
josehu07 Oct 19, 2023
edc62e5
update modeling script
josehu07 Oct 20, 2023
6f725d8
update modeling script
josehu07 Oct 20, 2023
8d01a79
staging progress on modeling scripts
josehu07 Oct 20, 2023
30aa677
update modeling scripts
josehu07 Oct 21, 2023
be38acd
update benchmarking scripts structure
josehu07 Oct 21, 2023
34766d1
add mess client for evaluation purposes
josehu07 Oct 22, 2023
0590641
change prob calculation pareto usage
josehu07 Oct 22, 2023
f3aceb2
fixing issues during benchmarking
josehu07 Oct 24, 2023
1bc77fb
fix gossip storm by introducing implicit Gossiped status
josehu07 Oct 24, 2023
078398e
fixing issues during benchmarking
josehu07 Oct 25, 2023
68e7b11
fixing issues during benchmarking
josehu07 Oct 25, 2023
fe6de0d
fixing issues during benchmarking
josehu07 Oct 25, 2023
8a98ea4
fixing issues during benchmarking
josehu07 Oct 25, 2023
803ac42
fix Raft duplicate reply bug and benmarking issues
josehu07 Oct 25, 2023
3eb5979
Merge branch 'main' into crossword
josehu07 Oct 25, 2023
138f34c
fix workflow_test.py local_clients command
josehu07 Oct 25, 2023
6abe6a8
Merge pull request #11 from josehu07/crossword
josehu07 Oct 25, 2023
a3021f2
better pause resume support in mess client
josehu07 Oct 25, 2023
c8a16ca
add promptly redirection upon pause
josehu07 Oct 25, 2023
a2bec2b
update modeling scripts
josehu07 Oct 26, 2023
605d2f0
fixed several deep bugs during benchmarking
josehu07 Oct 27, 2023
78182a5
fixing deep bugs during benchmarking
josehu07 Oct 28, 2023
430d77b
basically finished failover benchmark
josehu07 Oct 28, 2023
2f497ca
add netns & veth topology setup
josehu07 Oct 29, 2023
fd82000
better env setup & finished failover benchmark
josehu07 Oct 29, 2023
175be90
finish leader failover experiment
josehu07 Oct 29, 2023
a694df4
finish leader failover experiment
josehu07 Oct 29, 2023
e885d9f
skip unalive peers during gossiping selection
josehu07 Oct 31, 2023
c9f618c
update workflot_test.py
josehu07 Oct 31, 2023
0e6afca
Merge pull request #12 from josehu07/crossword
josehu07 Oct 31, 2023
4ad0e45
staging progress on adaptability
josehu07 Oct 31, 2023
69ff3fb
staging progress on adaptability
josehu07 Nov 1, 2023
ef097d7
staging progress on adaptability
josehu07 Nov 1, 2023
a578a13
staging progress on adaptability
josehu07 Nov 2, 2023
047ff61
staging progress on adaptability
josehu07 Nov 2, 2023
30f1f79
staging progress on adaptability
josehu07 Nov 3, 2023
bd4c398
staging progress on adaptability
josehu07 Nov 3, 2023
465bb9a
finished assignment policies management
josehu07 Nov 3, 2023
ad6eccc
fix Raft variants heartbeat slot bug
josehu07 Nov 3, 2023
e83c3e3
minor updates to plotting
josehu07 Nov 3, 2023
bdee260
finish leader failover experiment fixing
josehu07 Nov 3, 2023
63ac02e
refactoring protocol impls into submodules
josehu07 Nov 4, 2023
29846a0
decouple num shards from cluster size
josehu07 Nov 5, 2023
72f2fb7
minor updates
josehu07 Nov 5, 2023
69f0dad
implemented adaptability with linreg
josehu07 Nov 5, 2023
2b03fc8
finish adaptability implementation
josehu07 Nov 5, 2023
05c3f89
add support for value sizes over time to bench
josehu07 Nov 6, 2023
f9edbec
staging progress on bench adaptive
josehu07 Nov 6, 2023
201a2ed
staging progress on adaptability exper
josehu07 Nov 6, 2023
7a63f48
staging progress on adaptability exper
josehu07 Nov 7, 2023
d4b105a
staging progress on adaptability exper
josehu07 Nov 7, 2023
d8274fb
using ifb; finish unbalanced experiment
josehu07 Nov 7, 2023
ec7a0c5
finish runtime adaptability exper
josehu07 Nov 8, 2023
2d1ac39
staging progress on perf breakdown
josehu07 Nov 9, 2023
5eeb1aa
staging progress on perf breakdown
josehu07 Nov 9, 2023
67ac7dd
finish breakdown & space exper
josehu07 Nov 10, 2023
22081e1
staging progress on rs coding eval
josehu07 Nov 10, 2023
e7c7d10
finish rs coding overhead exper
josehu07 Nov 10, 2023
71d2356
minor updates to plotting scripts
josehu07 Nov 11, 2023
aa4ad9d
update perf over time figures
josehu07 Nov 12, 2023
76e5957
big update to adaptability figure
josehu07 Nov 12, 2023
416f6f7
minor updates to setup scripts
josehu07 Nov 12, 2023
44e734b
minor updates to plotting scripts
josehu07 Nov 12, 2023
3825bf6
minor updates to plotting scripts
josehu07 Nov 12, 2023
6775db1
minor updates to plotting scripts
josehu07 Nov 12, 2023
eab0934
Merge pull request #13 from josehu07/crossword
josehu07 Nov 12, 2023
d8951db
add pdf cropper helper script
josehu07 Nov 13, 2023
13bf371
add pdf cropper helper script
josehu07 Nov 13, 2023
767b1a6
Merge branch 'main' into crossword
josehu07 Nov 13, 2023
c8f5c9d
update pdf cropping script
josehu07 Nov 14, 2023
5b739a4
minor updates to plotting scripts
josehu07 Nov 14, 2023
1ffa223
minor updates to plotting scripts
josehu07 Nov 15, 2023
8788f2d
staging progress on critical path expers
josehu07 Nov 15, 2023
c19e23d
minor updates to plotting scripts
josehu07 Nov 16, 2023
c57e0a9
minor updates to plotting scripts
josehu07 Nov 16, 2023
60c6f0d
staging progress on critical path expers
josehu07 Nov 16, 2023
2c56f7a
remove unnecessary FillHoles & Commit messages
josehu07 Nov 17, 2023
de82498
almost finished critical path expers
josehu07 Nov 18, 2023
e045cce
finished critical path expers & plotting
josehu07 Nov 18, 2023
de08677
finished critical path expers & plotting
josehu07 Nov 18, 2023
999a3a7
minor updates to TLA+ spec
josehu07 Nov 19, 2023
0c14964
minor updates to plotting scripts
josehu07 Nov 19, 2023
3c40b00
minor updates to plotting scripts
josehu07 Nov 19, 2023
98801fa
fixed sim read lease bug
josehu07 Nov 19, 2023
103e2af
Merge pull request #14 from josehu07/crossword
josehu07 Nov 19, 2023
5dcd1f5
minor updates to plotting scripts
josehu07 Nov 20, 2023
5778db6
minor updates to plotting scripts
josehu07 Nov 20, 2023
20f4be8
minor updates to plotting scripts
josehu07 Nov 20, 2023
8ba8c42
add distributed machines helper scripts
josehu07 Nov 21, 2023
200c30f
minor updates to plotting scripts
josehu07 Nov 24, 2023
d9fff26
make toml dependency optional for workflow
josehu07 Nov 24, 2023
e54d8e0
Merge pull request #15 from josehu07/crossword
josehu07 Nov 24, 2023
b642172
staging progress on physical net expers
josehu07 Nov 24, 2023
0274ef1
add tidb workload profile results
josehu07 Nov 29, 2023
ea82157
staging progress on changed bench parameters
josehu07 Nov 30, 2023
e0de89d
update gitignore
josehu07 Nov 30, 2023
2cc7954
minor updates to tla+ formatting
josehu07 Nov 30, 2023
1e0f944
minor updates to plotting scripts
josehu07 Nov 30, 2023
513b9a6
minor updates to plotting scripts
josehu07 Nov 30, 2023
d14aaea
minor updates to plotting scripts
josehu07 Dec 1, 2023
a57b750
minor updates to plotting scripts
josehu07 Dec 1, 2023
fc55afc
add open_tcp_ports.sh script
josehu07 Dec 2, 2023
5299989
add auto iperf & ping script
josehu07 Dec 2, 2023
54d1a8b
staging progress on physical net expers
josehu07 Dec 2, 2023
71fc353
staging progress on physical net expers
josehu07 Dec 2, 2023
6ea626d
staging progress on physical net expers
josehu07 Dec 2, 2023
8490780
finished physical network exper =)
josehu07 Dec 3, 2023
4a52edb
finished physical network exper =)
josehu07 Dec 3, 2023
70ebb9b
Merge pull request #16 from josehu07/crossword
josehu07 Dec 3, 2023
a7045d3
update motivation profiling plots
josehu07 Dec 4, 2023
d66c211
add banal formal dump script
josehu07 Dec 5, 2023
495f4d6
minor updates to plotting scripts
josehu07 Dec 7, 2023
b2bdac1
Merge pull request #17 from josehu07/crossword
josehu07 Dec 7, 2023
bae3b4d
add WAN delays modeling calculation
josehu07 Feb 5, 2024
1288266
minor updates to WAN perf modeling script
josehu07 Feb 7, 2024
1eed1d4
reorganize scripts into paper-specific folders
josehu07 Feb 7, 2024
304bde0
reorganize scripts into paper-specific folders
josehu07 Feb 7, 2024
d303dde
mv artifact memos to separate file
josehu07 Feb 7, 2024
be2cb63
adding SMR-style TLA+ spec
josehu07 Feb 18, 2024
8b68ec8
adding command line TLC helper scripts
josehu07 Feb 18, 2024
15753e4
optimizing SMR-style TLA+ spec & linearizability constraint
josehu07 Feb 19, 2024
290ac9c
completing SMR-style TLA+ spec
josehu07 Feb 19, 2024
81e172c
completing SMR-style TLA+ spec
josehu07 Feb 19, 2024
228b17e
complete SMR-style MultiPaxos TLA+ spec
josehu07 Feb 20, 2024
715ba17
add README to MultiPaxos spec folder
josehu07 Feb 20, 2024
671ed88
add SMR-style MultiPaxos spec blog
josehu07 Feb 20, 2024
a45972a
minor updates to MultiPaxos spec README
josehu07 Feb 20, 2024
69c9902
update SMR-style MultiPaxos spec and TLC script
josehu07 Feb 21, 2024
52fbb95
add node failure injection to MultiPaxos spec
josehu07 Feb 21, 2024
7b94a25
update Crossword TLA+ spec
josehu07 Feb 21, 2024
c8e12ab
update Crossword TLA+ spec
josehu07 Feb 22, 2024
94ce043
update Crossword TLA+ spec .gitignore
josehu07 Feb 22, 2024
327eb25
extending SMR-style MultiPaxos spec
josehu07 Feb 23, 2024
1cb4569
extending SMR-style MultiPaxos spec
josehu07 Feb 24, 2024
b8ecce0
finish extended SMR-style MultiPaxos spec
josehu07 Feb 24, 2024
f71d792
staging progress on Bodega TLA+ spec
josehu07 Feb 26, 2024
928c365
staging progress on Bodega TLA+ spec
josehu07 Feb 27, 2024
3c273b9
finish Bodega TLA+ spec
josehu07 Feb 27, 2024
52e5bc7
finish Bodega TLA+ spec
josehu07 Feb 27, 2024
5ca31df
finish Bodega TLA+ spec
josehu07 Feb 27, 2024
67dbac1
finish Bodega TLA+ spec
josehu07 Feb 27, 2024
6b3701a
test change
josehu07 Feb 28, 2024
3cc2163
scanning through the codebase
josehu07 Mar 3, 2024
2b4089b
scanning through the codebase
josehu07 Mar 4, 2024
e85004a
add back committed condition check in Bodega spec
josehu07 Mar 7, 2024
728bcc4
cleaning up bench client
josehu07 Mar 18, 2024
d87ad40
add comments on port usage mess
josehu07 Mar 22, 2024
f9bbecd
remove useless perf sim parameters
josehu07 Mar 22, 2024
d821b96
minor updates to README
josehu07 Mar 22, 2024
0ef654b
add server module sync APIs
josehu07 Mar 23, 2024
ce895c2
fix wal_offset interference bug
josehu07 Mar 23, 2024
b76e863
refactor MultiPaxos code to fix Prepare phase bug
josehu07 Mar 24, 2024
7d03963
comment out the snapshotting test
josehu07 Mar 24, 2024
fd5800b
fixing the Prepare phase for RSPaxos
josehu07 Mar 24, 2024
22df5a1
fixing the Prepare phase for RSPaxos
josehu07 Mar 24, 2024
d175102
fixed the Prepare phase for RSPaxos
josehu07 Mar 25, 2024
4abc2e2
fixed the Prepare phase for Crossword
josehu07 Mar 25, 2024
fdd7d05
fix formatting issues
josehu07 Mar 25, 2024
4ac37d5
updating CloudLab experiments support
josehu07 Mar 28, 2024
b618a87
updating CloudLab experiments support
josehu07 Mar 28, 2024
be982e3
updating CloudLab experiments support
josehu07 Mar 28, 2024
6250a68
updating CloudLab experiments support
josehu07 Mar 28, 2024
f009e6d
updating CloudLab experiments support
josehu07 Mar 28, 2024
33123df
updating CloudLab experiments support
josehu07 Mar 29, 2024
b1aa6c4
updating CloudLab experiments support
josehu07 Mar 29, 2024
669843b
updating CloudLab experiments support
josehu07 Mar 29, 2024
85441de
updating CloudLab experiments support
josehu07 Mar 29, 2024
8ae312a
add remote_hosts.toml grouping
josehu07 Mar 29, 2024
796f785
restructured scripts utils library
josehu07 Mar 31, 2024
258c964
making better benchmarking scripts
josehu07 Mar 31, 2024
fd7c20f
making better benchmarking scripts
josehu07 Apr 3, 2024
f4ef176
making better benchmarking scripts
josehu07 Apr 5, 2024
29d2cb3
making better benchmarking scripts
josehu07 Apr 7, 2024
522b965
making better benchmarking scripts
josehu07 Apr 7, 2024
c4eab2f
making better benchmarking scripts
josehu07 Apr 8, 2024
c6e43b1
making better benchmarking scripts
josehu07 Apr 10, 2024
73d2732
finish follower read stalenss exper
josehu07 Apr 11, 2024
ba405d8
enable keyspace partitioning run mode
josehu07 Apr 12, 2024
34cef99
enable keyspace partitioning for ChainPaxos
josehu07 Apr 13, 2024
086dfdd
enable keyspace partitioning for ChainPaxos
josehu07 Apr 13, 2024
f72850f
enable keyspace partitioning for ChainPaxos
josehu07 Apr 13, 2024
db603d9
enable keyspace partitioning for ChainPaxos
josehu07 Apr 13, 2024
14c0139
minor updates to artifact readme
josehu07 Apr 15, 2024
45b84a5
working on improved YCSB exper
josehu07 Apr 16, 2024
2dd627a
working on improved YCSB exper
josehu07 Apr 16, 2024
220609e
finished improved YCSB trace exper
josehu07 Apr 16, 2024
6331fcc
finished improved YCSB trace exper
josehu07 Apr 16, 2024
a8908d8
update Crossword publish crop scripts
josehu07 Apr 17, 2024
4383b58
wrapping up the improved evaluation
josehu07 Apr 18, 2024
8eca21f
update results archiving script
josehu07 Apr 19, 2024
9bf7a98
add wan quorums model plotting for Bodega
josehu07 Apr 23, 2024
ecb167c
Merge branch 'bodega' of github.com:josehu07/summerset-private into b…
josehu07 Apr 23, 2024
9974387
use k reqs unit for ycsb plot
josehu07 Apr 24, 2024
32b9487
staging progress on chain replication impl
josehu07 Apr 24, 2024
4b8f1e4
Merge branch 'bodega' of github.com:josehu07/summerset-private into b…
josehu07 Apr 24, 2024
b13516d
staging progress on chain rep impl
josehu07 Apr 28, 2024
f8085bb
finished chain replication implementation
josehu07 Apr 29, 2024
1d7a192
finished chain replication implementation
josehu07 Apr 29, 2024
79e9b95
finished chain replication implementation
josehu07 Apr 29, 2024
e67947a
fixing CI workflow issues
josehu07 Apr 29, 2024
b28855b
Merge branch 'bodega'
josehu07 Apr 29, 2024
40535fc
fixing CI workflow issues
josehu07 Apr 29, 2024
5187a43
merge private main
josehu07 Apr 29, 2024
1b56ac6
merge private main
josehu07 Apr 29, 2024
986fd42
merge private main
josehu07 Apr 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions scripts/workflow_test.py → .github/workflow_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ def do_cargo_build():


def run_process(cmd):
# print("Run:", " ".join(cmd))
print("Run:", " ".join(cmd))
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
return proc

Expand Down Expand Up @@ -54,8 +54,11 @@ def wait_cluster_setup(proc, num_replicas):
for line in iter(proc.stderr.readline, b""):
l = line.decode()
print(l, end="", file=sys.stderr)
if "manager" not in l and "accepting clients" in l:
replica = int(l[l.find("(") + 1 : l.find(")")])
if "accepting clients" in l:
replica = l[l.find("(") + 1 : l.find(")")]
if replica == "m":
continue
replica = int(replica)
assert not accepting_clients[replica]
accepting_clients[replica] = True

Expand All @@ -66,7 +69,7 @@ def wait_cluster_setup(proc, num_replicas):
def run_tester_client(protocol, test_name):
cmd = [
"python3",
"./scripts/local_client.py",
"./scripts/local_clients.py",
"-p",
protocol,
"tester",
Expand All @@ -85,7 +88,7 @@ def run_tester_client(protocol, test_name):

do_cargo_build()

kill_all_matching("local_client.py", force=True)
kill_all_matching("local_clients.py", force=True)
kill_all_matching("local_cluster.py", force=True)
kill_all_matching("summerset_client", force=True)
kill_all_matching("summerset_server", force=True)
Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/tests_proc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ jobs:

steps:
- uses: actions/checkout@v3
- name: Install 'toml' package
run: pip3 install toml
- name: Run proc tests (MultiPaxos)
run: python3 scripts/workflow_test.py -p MultiPaxos
run: python3 .github/workflow_test.py -p MultiPaxos
- name: Run proc tests (Raft)
run: python3 scripts/workflow_test.py -p Raft
run: python3 .github/workflow_test.py -p Raft
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,13 @@
**/*.rs.bk
*.pdb

**/__pycache__/

/.vscode/

/tla+/**/states/
/tla+/**/*.out
/tla+/**/*.old

/results/
/backups/
Loading
Loading