Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multi_raft_test with L50 fails #362

Closed
akiradeveloper opened this issue May 22, 2024 · 0 comments · Fixed by #379
Closed

multi_raft_test with L50 fails #362

akiradeveloper opened this issue May 22, 2024 · 0 comments · Fixed by #379
Assignees
@akiradeveloper
Labels
bug Something isn't working

Comments

@akiradeveloper
Copy link
Owner

akiradeveloper commented May 22, 2024 •
edited
Loading

The test results in failure due to network error.

Error: status: Cancelled, message: "h2 protocol error: http2 error: stream error received: stream no longer needed", details: [], metadata: MetadataMap { headers: {} }

Caused by:
    0: transport error
    1: http2 error: stream error received: stream no longer needed
    2: stream error received: stream no longer needed

Before ending the test, re-election started due to network failure.

  lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 2
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 2
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 2
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 2
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 2
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 2
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:52Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 2
   lol-testapp-2|[2024-05-22T11:01:53Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:53Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 5
   lol-testapp-2|[2024-05-22T11:01:53Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:53Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 3
   lol-testapp-2|[2024-05-22T11:01:53Z INFO  lolraft::process::thread::election] election timeout. try to become a leader
   lol-testapp-2|[2024-05-22T11:01:53Z INFO  lolraft::process::voter::election] start pre-vote. try promote at term 2

However, I don't know what causes this. My machine has 32 vcpus and during the test the cpu usage didn't saturate but 90% in total (each core 3% or around). So the high cpu load is less likely the cause of the network failure.

Because the CPU has so much room, I guess it should be possible to have 1000 shards. Failing with 50 is unreasonable.

Maybe it is because of docker because docker network is notorious. I will investigate from this angle too.
@akiradeveloper akiradeveloper added the bug Something isn't working label May 22, 2024
@akiradeveloper akiradeveloper self-assigned this May 22, 2024
@akiradeveloper akiradeveloper linked a pull request May 25, 2024 that will close this issue
Change test params #379
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@akiradeveloper akiradeveloper

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Change test params akiradeveloper/sorock
1 participant
@akiradeveloper