Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

xenon rejoin xenon cluster is INVALID #144

Open
deep-cds opened this issue Jun 26, 2021 · 2 comments
Open

xenon rejoin xenon cluster is INVALID #144

deep-cds opened this issue Jun 26, 2021 · 2 comments
Assignees
@caphash

Comments

@deep-cds
Copy link

deep-cds commented Jun 26, 2021 •
edited
Loading

你好 、 xenon 研发同学。
部署好的 xenon+mysql 集群已经运行了一段时间， 发现执行 /data/xenon/bin/xenoncli cluster mysql 命令会出现cluster.go:480: unexpected error: get.client.error[dial tcp 192.168.39.167:8801: i/o timeout],如下图所示。
image

但是过一会执行命令 ，出现 如下情况
image

后面我把这个 xenon 程序stop ，然后start 。发现xenon的状态是 INVALID

image

以下是日志
[16:02:13 root@k8s-node-01 xenon]# cat xenon.log
xenon:[{Tag:v1.0.9-1-gc50a100 Time:2020/04/29 08:46:43 Git:c50a100 GoVersion:go1.14.2 Platform:linux amd64}]
2021/06/26 16:00:42.380592 [INFO] main: tag=[v1.0.9-1-gc50a100], git=[c50a100], goversion=[go1.14.2], builddate=[2020/04/29 08:46:43]
2021/06/26 16:00:42.380790 [WARNING] xenon.conf.raft:[&{MetaDatadir:raft.meta HeartbeatTimeout:1000 AdmitDefeatHtCnt:10 ElectionTimeout:3000 PurgeBinlogInterval:300000 SuperIDLE:false LeaderStartCommand: LeaderStopCommand: PurgeBinlogDisabled:false RequestTimeout:500 CandidateWaitFor2Nodes:60000}]
2021/06/26 16:00:42.380826 [WARNING] xenon.conf.mysql:[&{Admin:root Passwd:DOVYJ2gveVIirLzE Host:127.0.0.1 Port:3306 Basedir:/data/mysql8.0.22 Version:mysql57 DefaultsFile:/data/mysql8.0.22/config/my.cnf PingTimeout:1000 AdmitDefeatPingCnt:2 MasterSysVars: SlaveSysVars: ReplHost:192.168.39.167 ReplUser:repl ReplPasswd:replication}]
2021/06/26 16:00:42.380866 [WARNING] xenon.conf.mysqld:[&{SSHHost:192.168.39.167 SSHUser:root SSHPasswd:nnSnRp+xBKOFmSjF SSHPort:22 BackupDir:/data/mysql8.0.22/data XtrabackupBinDir:/usr/bin BackupIOPSLimits:100000 UseMemory:2GB Parallel:2 MysqldMonitorInterval:1000 Admin:root Passwd:DOVYJ2gveVIirLzE Host:127.0.0.1 Port:3306 Basedir:/data/mysql8.0.22 DefaultsFile:/data/mysql8.0.22/config/my.cnf}]
2021/06/26 16:00:42.381152 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].prepare.to.recovery.peers.from.[raft.meta].peers[[192.168.39.167:8801]].idlePeers[[]]
2021/06/26 16:00:42.381177 [INFO] server.prepare.setup.mysqlserver
2021/06/26 16:00:42.381186 [WARNING] mysqld.prepare.to.start...
2021/06/26 16:00:42.474056 [WARNING] mysqld.already.running...
2021/06/26 16:00:42.474078 [INFO] server.mysqlserver.setup.done
2021/06/26 16:00:42.474088 [INFO] server.mysql.wait.for.work[maxwait:60s]
2021/06/26 16:00:42.489500 [INFO] mysql.slave.status:&{mysql-bin.000004 57680655 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1958 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1958,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:00:42.489553 [INFO] server.mysql.gtid:{Master_Log_File:mysql-bin.000004 Read_Master_Log_Pos:57680655 Relay_Master_Log_File: Slave_IO_Running:true Slave_IO_Running_Str:Yes Slave_SQL_Running:true Slave_SQL_Running_Str:Yes Retrieved_GTID_Set:0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1958 Executed_GTID_Set:0cba804d-9845-11eb-b0db-fa163e9bad91:1-1958,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 Seconds_Behind_Master:0 Slave_SQL_Running_State:Slave has read all relay log; waiting for more updates Last_Error: Last_IO_Error: Last_SQL_Error:}
2021/06/26 16:00:42.489575 [INFO] server.mysql.set.to.READONLY
2021/06/26 16:00:42.490123 [INFO] server.mysql.start.slave
2021/06/26 16:00:42.490340 [INFO] server.mysql.check.replication.user...
2021/06/26 16:00:42.490686 [INFO] server.mysql.setup.done
2021/06/26 16:00:42.490701 [INFO] server.prepare.setup.RPC
2021/06/26 16:00:42.491009 [INFO] server.RPC.setup.done
2021/06/26 16:00:42.491036 [INFO] mysqld[/data/mysql8.0.22/config/my.cnf].monitor.start...
2021/06/26 16:00:42.491047 [INFO] mysql[127.0.0.1:3306].startping...
2021/06/26 16:00:42.491061 [INFO] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].raft.start...
2021/06/26 16:00:42.491175 [INFO] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].start.CheckBrainSplit
2021/06/26 16:00:42.491206 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].state.init
2021/06/26 16:00:42.491326 [WARNING] xrpc.Start.listening.on[192.168.39.167:8801]
2021/06/26 16:00:42.491337 [INFO] server.start.success...
2021/06/26 16:00:42.491343 [INFO] xenon.start.success...
2021/06/26 16:00:42.494777 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].leaderStopShellCommand[[-c ]].done
2021/06/26 16:00:42.494803 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].mysql.waitMysqlDoneAsync.prepare
2021/06/26 16:00:42.494842 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].state.machine.run
2021/06/26 16:00:42.495409 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].mysql.SetReadOnly.done
2021/06/26 16:00:42.495427 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].mysql.SetSlaveGlobalSysVar.done
2021/06/26 16:00:42.495436 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].prepareAsync.done
2021/06/26 16:00:42.501336 [INFO] mysql.slave.status:&{mysql-bin.000004 57680655 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1958 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1958,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:00:42.501398 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].init.my.gtid.is:{mysql-bin.000004 57680655 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1958 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1958,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:00:43.459841 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:43.991472 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:44.459370 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:45.459558 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:45.491951 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:46.460700 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:46.992236 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:47.462440 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:48.459498 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:48.492406 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:49.459956 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:49.992834 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:50.458966 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:51.463800 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:51.493678 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:52.459795 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:52.993987 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:53.460034 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:54.457774 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:54.494107 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:55.460327 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:55.994357 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:56.463465 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:57.462650 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:57.494515 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:58.462458 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:00:58.994784 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:00:59.460173 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:00.460270 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:00.494962 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:01.461057 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:01.995114 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:02.462786 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:03.463594 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:03.495539 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:04.463260 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:04.995809 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:05.462384 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:06.464705 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:06.495961 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:07.463703 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:07.996260 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:08.463654 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:09.464480 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:09.496800 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:10.460974 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:10.997015 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:11.463696 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:12.465038 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:12.497577 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:13.465436 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:13.997873 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:14.467418 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:15.463660 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:15.498224 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:16.464437 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:16.998408 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:17.464747 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:18.469714 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:18.499186 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:19.465306 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:19.999382 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:20.464325 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:21.463087 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:21.500581 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:22.464104 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:23.000820 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:23.466962 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:24.467790 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:24.501302 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:24.570185 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:25.466976 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:25.666384 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:26.001421 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:26.226163 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:26.466934 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:26.816453 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:27.349656 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:27.466789 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:27.501571 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:27.847386 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:28.467647 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:29.002103 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:29.467825 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:30.467635 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:30.503307 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:31.468236 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:32.003609 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:32.468363 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:33.469618 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:33.503849 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:34.472446 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:35.004252 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:35.470297 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:36.470724 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:36.505045 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:37.471026 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:38.005288 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:38.470004 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:39.470284 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:39.506212 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:40.469646 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:41.006448 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:41.471915 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:42.474189 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:42.506759 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:43.103250 [INFO] mysql.slave.status:&{mysql-bin.000004 57685766 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1964 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1964,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:43.472371 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:44.007034 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:44.474044 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:45.399258 [INFO] mysql.slave.status:&{mysql-bin.000004 57685766 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1964 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1964,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:45.472811 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:45.507228 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:46.338145 [INFO] mysql.slave.status:&{mysql-bin.000004 57685766 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1964 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1964,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:46.473672 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:47.007541 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:47.473403 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:47.968206 [INFO] mysql.slave.status:&{mysql-bin.000004 57685766 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1964 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1964,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:48.473309 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:48.507730 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:48.797818 [INFO] mysql.slave.status:&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:49.475140 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:50.007921 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:50.008067 [INFO] mysql.slave.status:&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:50.473337 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:50.832914 [INFO] mysql.slave.status:&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:01:51.472657 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:51.508092 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:52.475658 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:53.008452 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:53.474967 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:54.473878 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:54.508610 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:55.474622 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:56.008813 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:56.475106 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:57.476667 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:57.509003 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:58.474316 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:01:59.009338 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:01:59.476117 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:02:00.474590 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:02:00.509589 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:02:01.473740 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:02:02.009840 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:02:02.476089 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:02:03.475916 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:02:03.509956 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:02:04.476939 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:02:05.010190 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:02:05.492110 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:02:06.477866 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest
2021/06/26 16:02:06.510434 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:02:07.321738 [WARNING] server.rpc.node.add:&{From: Nodes:[192.168.39.162:8802 192.168.39.127:8803]}
2021/06/26 16:02:07.322225 [INFO] FOLLOWER[ID:192.168.39.167:8801, V:0, E:1].check.peers.json.file[raft.meta/peers.json].stat[name:peers.json, mode:-rwxr-xr-x, size:72, lastmodified:2021-06-26 16:02:07.321878393 +0800 CST]
2021/06/26 16:02:07.322280 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:1].add.peer[192.168.39.162:8802].to.peers[map[192.168.39.162:8802:0xc0001895c0]]
2021/06/26 16:02:07.329151 [INFO] FOLLOWER[ID:192.168.39.167:8801, V:0, E:2].check.peers.json.file[raft.meta/peers.json].stat[name:peers.json, mode:-rwxr-xr-x, size:94, lastmodified:2021-06-26 16:02:07.328878364 +0800 CST]
2021/06/26 16:02:07.329185 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:2].add.peer[192.168.39.127:8803].to.peers[map[192.168.39.127:8803:0xc000189710 192.168.39.162:8802:0xc0001895c0]]
2021/06/26 16:02:07.487994 [INFO] mysql.slave.status:&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:02:07.488038 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:2].get.heartbeat.my.gtid.is:{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:02:07.546353 [INFO] mysql.get.local.uuid:[8b92218c-af9f-11eb-8207-fa163e0362cf]
2021/06/26 16:02:07.546920 [INFO] mysql.get.local.uuid:[8b92218c-af9f-11eb-8207-fa163e0362cf]
2021/06/26 16:02:07.547729 [WARNING] follower.gtid[&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }].bigger.than.remote[&{mysql-bin.000004 57686187 true true 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77658,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 }]
2021/06/26 16:02:07.547762 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].get.heartbeat.from[N:192.168.39.162:8802, V:105, E:2].change.mysql.master
2021/06/26 16:02:08.010688 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].ping.responses[1].is.less.than.half.maybe.brain.split
2021/06/26 16:02:08.013100 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].send.ping.to.peer[192.168.39.127:8803].client.call.ok.rsp[&{{0 0 FOLLOWER} { 0 false false } OK}]
2021/06/26 16:02:08.014062 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].send.ping.to.peer[192.168.39.162:8802].client.call.ok.rsp[&{{0 0 LEADER} { 0 false false } OK}]
2021/06/26 16:02:08.394047 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].get.heartbeat.change.to.the.new.master[192.168.39.162:8802].successed
2021/06/26 16:02:08.394078 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].get.heartbeat.from[N:192.168.39.162:8802, V:105, E:2].update.view
2021/06/26 16:02:08.394089 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].do.updateViewID[FROM:0 TO:105]
2021/06/26 16:02:08.394458 [WARNING] INVALID[ID:192.168.39.167:8801, V:105, E:2].follower.state.machine.exit
2021/06/26 16:02:08.397859 [WARNING] INVALID[ID:192.168.39.167:8801, V:105, E:2].leaderStopShellCommand[[-c ]].done
2021/06/26 16:02:11.411235 [INFO] mysql.slave.status:&{mysql-bin.000004 57686633 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1966,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:02:13.316226 [INFO] mysql.slave.status:&{mysql-bin.000004 57686633 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1966,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:02:20.152371 [INFO] mysql.slave.status:&{mysql-bin.000004 57687054 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966-1967 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1967,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:02:21.160871 [INFO] mysql.slave.status:&{mysql-bin.000004 57687054 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966-1967 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1967,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }
2021/06/26 16:02:21.740884 [INFO] mysql.slave.status:&{mysql-bin.000004 57687054 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966-1967 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1967,
8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660,
933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793,
eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }

请xenon 研发人员帮忙看一下。 谢谢
@caphash
Copy link

caphash commented Jun 21, 2022

你好，问题可以分为两个部分解答：

  1. xenoncli 执行 timeout：

通过以下日志可以看到 xenon 配置 RequestTimeout 为 0.5 秒过小，在通常的网络条件下建议不小于 1 秒 并设置为 ElectionTimeout/AdmitDefeatHtCnt 的值。

2021/06/26 16:00:42.380790 [WARNING] xenon.conf.raft:[&{MetaDatadir:raft.meta HeartbeatTimeout:1000 AdmitDefeatHtCnt:10 ElectionTimeout:3000 PurgeBinlogInterval:300000 SuperIDLE:false LeaderStartCommand: LeaderStopCommand: PurgeBinlogDisabled:false RequestTimeout:500 CandidateWaitFor2Nodes:60000}] 2021/06/26 16:00:42.380826 [WARNING] xenon.conf.mysql:[&{Admin:root Passwd:DOVYJ2gveVIirLzE Host:127.0.0.1 Port:3306 Basedir:/data/mysql8.0.22 Version:mysql57 DefaultsFile:/data/mysql8.0.22/config/my.cnf PingTimeout:1000 AdmitDefeatPingCnt:2 MasterSysVars: SlaveSysVars: ReplHost:192.168.39.167 ReplUser:repl ReplPasswd:replication}] 2021/06/26 16:00:42.380866 [WARNING] xenon.conf.mysqld:[&{SSHHost:192.168.39.167 SSHUser:root SSHPasswd:nnSnRp+xBKOFmSjF SSHPort:22 BackupDir:/data/mysql8.0.22/data XtrabackupBinDir:/usr/bin BackupIOPSLimits:100000 UseMemory:2GB Parallel:2 MysqldMonitorInterval:1000 Admin:root Passwd:DOVYJ2gveVIirLzE Host:127.0.0.1 Port:3306 Basedir:/data/mysql8.0.22 DefaultsFile:/data/mysql8.0.22/config/my.cnf}]

  1. 主从发生过切换且原主 192.168.39.167 状态变为 INVALID：

这是由于主节点写数据时切换，leader 在降级后存在脏事务，当 xenon 检测到时自动将原主设置为 INVALID 状态，出现该情况后可以通过 ./xenoncli mysql rebuildme 进行数据重建，之后恢复为 FOLLOWER 状态。

另外，可以适当调大 HeartbeatTimeout、ElectionTimeout 等参数，并使用 xenon 最新版降低误切换可能性。

@caphash caphash self-assigned this Jun 21, 2022
@Lzw901
Copy link

Lzw901 commented Apr 15, 2024

[mysql@localhost /usr/local/xenon/bin]$ ./xenoncli mysql rebuildme --from=10.232.2.187:8801
2024/04/15 17:46:19.903706 [WARNING] =====prepare.to.rebuildme=====
IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".

2024/04/15 17:46:19.904072 [WARNING] S1-->check.raft.leader
2024/04/15 17:46:19.906042 [WARNING] S2-->prepare.rebuild.from[10.232.2.187:8801]....
2024/04/15 17:46:19.908912 [PANIC] I[10.232.2.188:8801].have.[2].local.transactions.more.than.maxAllowedLocalTrxCount[0].compared.to.from[10.232.2.187:8801].you.cant.rebuildme.sir
panic: [PANIC] I[10.232.2.188:8801].have.[2].local.transactions.more.than.maxAllowedLocalTrxCount[0].compared.to.from[10.232.2.187:8801].you.cant.rebuildme.sir

goroutine 1 [running]:
xbase/xlog.(*Log).Panic(0xc0001862e0, {0x7b2085, 0x7fffa76da2da}, {0xc000121db0, 0xc000190750, 0x0})
/root/xenon-1.1.4/src/xbase/xlog/xlog.go:142 +0xc8
cli/cmd.mysqlRebuildMeCommandFn(0xc0001e6b40, {0xc000186370, 0x1, 0x1})
/root/xenon-1.1.4/src/cli/cmd/mysql.go:269 +0x438
vendor/github.com/spf13/cobra.(*Command).execute(0xc0001e6b40, {0xc000186350, 0x1, 0x1})
/root/xenon-1.1.4/src/vendor/github.com/spf13/cobra/command.go:603 +0x3a7
vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xa5b720)
/root/xenon-1.1.4/src/vendor/github.com/spf13/cobra/command.go:689 +0x24b
vendor/github.com/spf13/cobra.(*Command).Execute(...)
/root/xenon-1.1.4/src/vendor/github.com/spf13/cobra/command.go:648
main.main()
/root/xenon-1.1.4/src/cli/cli.go:43 +0x25

重建节点时报错这个是什么原因？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@caphash caphash

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@deep-cds @caphash @Lzw901