support download flushed binlog and parse event for cloud comput…

[BLAZZ]

in cloud computing paltform, binlog may flush to cloud storage, that will cause 'not find first log', this change support to consume the position from a flushed binlog

for aliyun example

type RDSBinlogDownloader struct {
	Endpoint   string
	AccessKey  string
	KeySecret  string
	InstanceId string
	UsePubNet  bool
}

func (r *RDSBinlogDownloader) DownloaderFunc() canal.BinlogFileDownload {
	return func(pos mysql.Position) (path string, err error) {
		return r.downBinlogFile(pos.Name)
	}
}

func (r *RDSBinlogDownloader) downBinlogFile(fileName string) (string, error) {
	//	call rds api to find binlog file by name and download it to local filesytem
	...
}

use RDSBinlogDownloader

var current *model.Position // get from myapp's position storage
var canalErr error
if current == nil {
	if t.rdsDownload != nil {
		t.canal.WithLocalBinlogDownload(t.rdsDownload)
	}
	canalErr = t.canal.Run()
} else {
	myPos := mysql.Position{
		Name: current.Name,
		Pos:  current.Pos,
	}
	if t.rdsDownload != nil {
		t.canal.WithLocalBinlogDownload(t.rdsDownload)
	}
	canalErr = t.canal.RunFrom(myPos)
}

[lance6716]

If the binlog file is purged just at the time we are switching to it, should we also generate a "fake rotate evnet" as the real binlog streamer?

https://github.com/mysql/mysql-server/blob/4f1d7cf5fcb11a3f84cff27e37100d7295e7d5ca/sql/rpl_binlog_sender.cc#L248

[BLAZZ]

if syncer switch to a purged binlog file, 'not find first log file' happen and syncer closed. it try to download binlog to local. the local file contains rotate event, and will put the rotate event to streamer. then canal's masterInfo will change to the new position and it restart syncer to try if new position file is on master, if not the new started syncer will also closed by 'not find first log file' and download next binlog to local. so generate a "fake rotate evnet" maybe not necessary?

[lance6716]

I mean the "fake rorate event" which comes before the beginning of any events in binlog, not the real rotate event at the end of each binlog file. You can find it when reading binlog

=== RotateEvent ===
Date: 2022-12-08 11:13:59
Log position: 1969
Event size: 47
Position: 4
Next log name: mysql-bin.000002

=== RotateEvent ===
Date: 1970-01-01 08:00:00
Log position: 0
Event size: 47
Position: 4
Next log name: mysql-bin.000002

for above two events, the first one is the real rotate event which can be found at the end of mysql-bin.000001, and the second one is a fake rotate event

[BLAZZ]

"fake rorate event" not exist in binlog file, so no need to process?

I find this in mariadb docs

[lance6716]

"fake rorate event" does not exist in binlog file but it is sent to downstream and can be find when reading binlog 😅

[BLAZZ]

when using local binlog, the real syncer is closed, canal will not read event from server's binlog, all event read from AdapterStreamer is parsed from binlog file, that will not exist "fake rorate event". "fake rorate event" will send to streamer after finish parse binlog file and restart the read syncer, and runSyncBinlog will process that event

[lance6716]

do we need to add an error that no matching position for position.Pos? For example user downlaods wrong binlog file

[lance6716]

can return err to let caller streamer.CloseWithError(err) in line 112

[lance6716]

so we don't check it's a GTID-based replication? For GTID-based replication, filename and position is not dependable to reset replication because upstream master may failover to a new MySQL node and they have same binlog filename but different position.

[lance6716]

"fake rorate event" does not exist in binlog file but it is sent to downstream and can be find when reading binlog 😅

[Danny5487401]

any progress for this support?

[BLAZZ]

any progress for this support?

you may try gitee.com/Blazz/go-mysql-plus.git.
our company has run for this feature in production for almost one year, it works OK.

PS: cloud computing platform will auto switch master to salver and raise local binlog file not exist error.
you should record the GTID and using 'SHOW BINLOG EVENTS IN xxx' to find the next position after canal.RunFrom stop and restart canal if found the next position

[dveeden]

@BLAZZ I would like to get this merged. Could you check the open conversations and fix the conflicts?