add throttle to prevent cpu load

[alkaaf]

i hope this is usable. i need this when resume from certain bin log position

[lance6716]

And can you explain your problem first? I guess you are using this library and other processes in the same machine, and this library comsumes too much CPU. But a more common way is to use cgroup or something to limit the resource of the process using this library.

[lance6716]

should use

select {
case <-time.After...
case <-b.ctx.Done...
}

to find context cancel timely

[alkaaf]

I have experience using maxwell for etl from mysql to mongodb where reading result from maxwell i store in redis queue. there are some workers consume queue to store it in mongodb

This approach has a lot of overhead from the network side and also expensive (redis, worker, maxwell that using java). therefore I need a cheaper, concise, reliable and consistent approach where delay is not an issue.

the ability to continue from the last log position was something I was looking for from this library. the problem is that when this app fails/stops some time and when restarting the app, the channel gets flooded with events starting from a certain position. it causes event processing to slow down to even stall (yeah you are right that i do event processing too on the same machine even application). this is why limiting resources is not an option.

therefore I need something to delay the binlog read by a few milliseconds when the channel reaches a certain size. (in my case, when the channel size is 512 ,the read will be delayed by 1 second)

[lance6716]

I'm still not very clear about your usage, but I'd ask do you think it's a good idea to add an option to control the channel size? So when channel is full, sending to channel will be blocked and no further CPU / network resources will be consumed.

go-mysql/replication/binlogstreamer.go

Lines 87 to 90 in dc97dfa

	func NewBinlogStreamer() *BinlogStreamer {
	s := new(BinlogStreamer)
	s.ch = make(chan *BinlogEvent, 10240)

Maybe you can set the channel size to 512, and the delay will not be as bad as 1 second.

[lance6716]

Hi, can you check #830 is helpful to your problem?

[dveeden]

@alkaaf is this still a problem for you? if so, could you fix the conflict?