Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task is ahead/behind of ShardState #98

Open
ckhan opened this issue Jul 18, 2016 · 0 comments
Open

Task is ahead/behind of ShardState #98

ckhan opened this issue Jul 18, 2016 · 0 comments

Comments

@ckhan
Copy link

ckhan commented Jul 18, 2016

We use mapreduce and pipelines for various administrative jobs on GAE.
Almost every day, I'll see warnings like this in the log:

LOG 2 1468861523504730 Task 1571100778565B9A65B43-5-0 is behind ShardState 1. Dropping task.
..
LOG 2 1468861525573810 Shard 1571100778626ECE727ED-7 is not active. Possible spurious task execution. Dropping this task.
..
LOG 2 1468861525628980 ShardState is {active:False,last_work_item:Key('FileEntity', 5711710446944256, namespace='5666083260334080'),result_status:success,slice_id:3,update_time:2016-07-18 17:04:00.927030,}
..
LOG 2 1468618964079700 Task 15711250257171C594209-3-2 is ahead of ShardState 1. Waiting for it to catch up.

They seem to be innocuous (just retries?), but they are making a lot of unwanted noise in the logs.
Questions:

  • what do these errors mean?
  • if it's innocuous, should it be logged as warning? should we drop logging level in our fork?
  • if not, what can we do to mitigate the underlying issue?

Thanks for any pointers!
-ck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant