NewBatchingHook doesn't seem to grab the sequence token

[ramonkeypr]

I am attempting to use batching to avoid CloudWatch rate limiting. During my automated testing, I am seeing the following:

=== RUN   TestCloudWatchBatching100
Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642
	status code: 400, request id: f15938de-5063-11e7-8263-adbc854f83ad
{"app":"middlewares.test","host":"45f28166fed1","level":"info","msg":"test-host-100 - username [ts] \"method uri proto\" %!d(string=status) %!d(string=size) \"referer\" \"agent\" fblh.reqid \"frontend\" \"backend\" elapsed\n","time":1497377582}Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642
	status code: 400, request id: f15e8fab-5063-11e7-bf73-6fbae9798f07
{"app":"middlewares.test","host":"45f28166fed1","level":"info","msg":"test-host-100 - username [ts] \"method uri proto\" %!d(string=status) %!d(string=size) \"referer\" \"agent\" fblh.reqid \"frontend\" \"backend\" elapsed\n","time":1497377582}Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642
	status code: 400, request id: f16482f0-5063-11e7-acda-e388958e2b61
{"app":"middlewares.test","host":"45f28166fed1","level":"info","msg":"test-host-100 - username [ts] \"method uri proto\" %!d(string=status) %!d(string=size) \"referer\" \"agent\" fblh.reqid \"frontend\" \"backend\" elapsed\n","time":1497377582}Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642
	status code: 400, request id: f168c988-5063-11e7-abdd-139e182e19f7
{"app":"middlewares.test","host":"45f28166fed1","level":"info","msg":"test-host-100 - username [ts] \"method uri proto\" %!d(string=status) %!d(string=size) \"referer\" \"agent\" fblh.reqid \"frontend\" \"backend\" elapsed\n","time":1497377582}Failed to fire hook: InvalidSequenceTokenException: The given sequenceToken is invalid. The next expected sequenceToken is: 49570318328963403756104352477615904793123053365436134642

/snip

If I can find a fix, I will submit a PR.

[kdar]

@punya-asapp would you know the issue here? I'm kind of swamped at the moment.

[punya-asapp]

I'll take a look tomorrow

[SteveHNH]

Not sure how relevant this is these days as this issue is pretty old, but I did a bunch of fiddling with the batching hook. It seems to work fine if it's the only process, but when you have multiple processes logging to the same logstream, you can end up in a state where the sequence token is incorrect and it never corrects itself.
The workaround is you either grab a new sequence token when it fails (retry), or you log to different log streams per process.
I have some retry logic I can put in a PR if it seems valuable, but I ended up just going with logging to a different stream per process as it was the most simple. The logs are aggregated in elasticsearch in our architecture so multiple streams doesn't really hurt.

[kdar]

Any PR is appreciated if you believe it's valuable.