Deploying locally failures.

[izaakdale]

I did a fair bit of hacking & hard coding to even get the init pod up and running, now that I have I get a bunch of error logs. It seems a leader is never actually allocated and elections happen over and over. I have also tried downloading the source code and running that, I get a similar log output from there. Has anyone faced this issue? As you can see from the logs leader is always blank.

I changed the config.yaml in statefulset to look like this:

ID=$(echo $HOSTNAME | rev | cut -d- -f1 | rev)
cat > /var/run/proglog/config.yaml <<EOD
data-dir: /var/run/proglog/data
rpc-port: {{.Values.rpcPort}}
bind-addr: "$HOSTNAME.proglog.{{.Release.Namespace}}.svc.cluster.local:{{.Values.serfPort}}"
bootstrap: $([ $ID = 0 ] && echo true || echo false)
$([ $ID != 0 ] && echo 'start-join-addrs: "proglog-0.proglog.{{.Release.Namespace}}.svc.cluster.local:{{.Values.serfPort}}"' )
EOD

Source code logs:

Defaulted container "proglog" out of: proglog, proglog-config-init (init)
2023-04-21T18:59:51.680Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:proglog-0 Address:proglog-0.proglog.default.svc.cluster.local:8400}]
2023-04-21T18:59:51.680Z [INFO]  raft: Node at 10.244.0.50:8400 [Follower] entering Follower state (Leader: "")
2023-04-21T18:59:52.801Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2023-04-21T18:59:52.801Z [INFO]  raft: Node at 10.244.0.50:8400 [Candidate] entering Candidate state in term 6
2023-04-21T18:59:52.810Z [DEBUG] raft: Votes needed: 1
2023-04-21T18:59:52.810Z [DEBUG] raft: Vote granted from proglog-0 in term 6. Tally: 1
2023-04-21T18:59:52.810Z [INFO]  raft: Election won. Tally: 1
2023-04-21T18:59:52.810Z [INFO]  raft: Node at 10.244.0.50:8400 [Leader] entering Leader state
2023-04-21T18:59:52.810Z [ERROR] raft: Failed to commit logs: EOF
2023-04-21T18:59:52.810Z [INFO]  raft: Node at 10.244.0.50:8400 [Follower] entering Follower state (Leader: "")
2023-04-21T18:59:53.856Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2023-04-21T18:59:53.857Z [INFO]  raft: Node at 10.244.0.50:8400 [Candidate] entering Candidate state in term 7
2023-04-21T18:59:53.873Z [DEBUG] raft: Votes needed: 1
2023-04-21T18:59:53.875Z [DEBUG] raft: Vote granted from proglog-0 in term 7. Tally: 1
2023-04-21T18:59:53.876Z [INFO]  raft: Election won. Tally: 1
2023-04-21T18:59:53.876Z [INFO]  raft: Node at 10.244.0.50:8400 [Leader] entering Leader state
2023-04-21T18:59:53.876Z [ERROR] raft: Failed to commit logs: EOF
2023-04-21T18:59:53.876Z [INFO]  raft: Node at 10.244.0.50:8400 [Follower] entering Follower state (Leader: "")
Error: timed out

My logs:

Defaulted container "logger-service-log" out of: logger-service-log, logger-service-log-config-init (init)
2023-04-21T18:39:39.614Z [INFO]  raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:logger-service-log-0 Address:logger-service-log.default.svc.cluster.local:8400}]"
2023-04-21T18:39:39.614Z [INFO]  raft: entering follower state: follower="Node at 10.244.0.48:8400 [Follower]" leader-address= leader-id=
2023/04/21 18:39:39 [INFO] serf: EventMemberJoin: logger-service-log-0 10.244.0.48
2023-04-21T18:39:40.662Z [WARN]  raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-04-21T18:39:40.662Z [INFO]  raft: entering candidate state: node="Node at 10.244.0.48:8400 [Candidate]" term=466
2023-04-21T18:39:40.665Z [DEBUG] raft: voting for self: term=466 id=logger-service-log-0
2023-04-21T18:39:40.671Z [DEBUG] raft: calculated votes needed: needed=1 term=466
2023-04-21T18:39:40.671Z [DEBUG] raft: vote granted: from=logger-service-log-0 term=466 tally=1
2023-04-21T18:39:40.671Z [INFO]  raft: election won: term=466 tally=1
2023-04-21T18:39:40.671Z [INFO]  raft: entering leader state: leader="Node at 10.244.0.48:8400 [Leader]"
2023-04-21T18:39:40.671Z [ERROR] raft: failed to commit logs: error=EOF
2023-04-21T18:39:40.671Z [INFO]  raft: entering follower state: follower="Node at 10.244.0.48:8400 [Follower]" leader-address= leader-id=
2023-04-21T18:39:41.672Z [WARN]  raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-04-21T18:39:41.672Z [INFO]  raft: entering candidate state: node="Node at 10.244.0.48:8400 [Candidate]" term=467
2023-04-21T18:39:41.675Z [DEBUG] raft: voting for self: term=467 id=logger-service-log-0
2023-04-21T18:39:41.681Z [DEBUG] raft: calculated votes needed: needed=1 term=467
2023-04-21T18:39:41.681Z [DEBUG] raft: vote granted: from=logger-service-log-0 term=467 tally=1
2023-04-21T18:39:41.681Z [INFO]  raft: election won: term=467 tally=1
2023-04-21T18:39:41.681Z [INFO]  raft: entering leader state: leader="Node at 10.244.0.48:8400 [Leader]"
2023-04-21T18:39:41.681Z [ERROR] raft: failed to commit logs: error=EOF
2023-04-21T18:39:41.681Z [INFO]  raft: entering follower state: follower="Node at 10.244.0.48:8400 [Follower]" leader-address= leader-id=
2023-04-21T18:39:42.688Z [WARN]  raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-04-21T18:39:42.688Z [INFO]  raft: entering candidate state: node="Node at 10.244.0.48:8400 [Candidate]" term=468
2023-04-21T18:39:42.692Z [DEBUG] raft: voting for self: term=468 id=logger-service-log-0
2023-04-21T18:39:42.697Z [DEBUG] raft: calculated votes needed: needed=1 term=468
2023-04-21T18:39:42.697Z [DEBUG] raft: vote granted: from=logger-service-log-0 term=468 tally=1
2023-04-21T18:39:42.698Z [INFO]  raft: election won: term=468 tally=1
2023-04-21T18:39:42.698Z [INFO]  raft: entering leader state: leader="Node at 10.244.0.48:8400 [Leader]"
2023-04-21T18:39:42.698Z [ERROR] raft: failed to commit logs: error=EOF
2023-04-21T18:39:42.698Z [INFO]  raft: entering follower state: follower="Node at 10.244.0.48:8400 [Follower]" leader-address= leader-id=
2023-04-21T18:39:44.379Z [WARN]  raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-04-21T18:39:44.380Z [INFO]  raft: entering candidate state: node="Node at 10.244.0.48:8400 [Candidate]" term=469
2023-04-21T18:39:44.383Z [DEBUG] raft: voting for self: term=469 id=logger-service-log-0
2023-04-21T18:39:44.390Z [DEBUG] raft: calculated votes needed: needed=1 term=469
2023-04-21T18:39:44.390Z [DEBUG] raft: vote granted: from=logger-service-log-0 term=469 tally=1
2023-04-21T18:39:44.390Z [INFO]  raft: election won: term=469 tally=1
2023-04-21T18:39:44.390Z [INFO]  raft: entering leader state: leader="Node at 10.244.0.48:8400 [Leader]"
2023-04-21T18:39:44.390Z [ERROR] raft: failed to commit logs: error=EOF
2023-04-21T18:39:44.390Z [INFO]  raft: entering follower state: follower="Node at 10.244.0.48:8400 [Follower]" leader-address= leader-id=
2023-04-21T18:39:45.862Z [WARN]  raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-04-21T18:39:45.873Z [INFO]  raft: entering candidate state: node="Node at 10.244.0.48:8400 [Candidate]" term=470
2023-04-21T18:39:45.966Z [DEBUG] raft: voting for self: term=470 id=logger-service-log-0
2023-04-21T18:39:45.983Z [DEBUG] raft: calculated votes needed: needed=1 term=470
2023-04-21T18:39:45.983Z [DEBUG] raft: vote granted: from=logger-service-log-0 term=470 tally=1
2023-04-21T18:39:45.983Z [INFO]  raft: election won: term=470 tally=1
2023-04-21T18:39:45.983Z [INFO]  raft: entering leader state: leader="Node at 10.244.0.48:8400 [Leader]"
2023-04-21T18:39:45.983Z [ERROR] raft: failed to commit logs: error=EOF
2023-04-21T18:39:45.983Z [INFO]  raft: entering follower state: follower="Node at 10.244.0.48:8400 [Follower]" leader-address= leader-id=
2023-04-21T18:39:47.023Z [WARN]  raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2023-04-21T18:39:47.023Z [INFO]  raft: entering candidate state: node="Node at 10.244.0.48:8400 [Candidate]" term=471
2023-04-21T18:39:47.027Z [DEBUG] raft: voting for self: term=471 id=logger-service-log-0
2023-04-21T18:39:47.033Z [DEBUG] raft: calculated votes needed: needed=1 term=471
2023-04-21T18:39:47.033Z [DEBUG] raft: vote granted: from=logger-service-log-0 term=471 tally=1
2023-04-21T18:39:47.033Z [INFO]  raft: election won: term=471 tally=1
2023-04-21T18:39:47.033Z [INFO]  raft: entering leader state: leader="Node at 10.244.0.48:8400 [Leader]"
2023-04-21T18:39:47.033Z [ERROR] raft: failed to commit logs: error=EOF
2023-04-21T18:39:47.033Z [INFO]  raft: entering follower state: follower="Node at 10.244.0.48:8400 [Follower]" leader-address= leader-id=

[cschar]

same issues encountered, trying to track here:
#11 (comment)

[Gh05ts]

Have found a partial, can probably be worked out for a proper fix under this issue
#11