Skip to content

Commit

Permalink
task/runner: Avoid re-running poison tasks (#20)
Browse files Browse the repository at this point in the history
We have some tasks that are being re-executed
over and over again since they get stuck in the
task-runner logic. We should fix the root cause
of those, but to avoid the problem from getting
worse we should also avoid re-running these tasks
over and over again.

This fixes that by not even starting tasks that we
find out had already been started (phase=running).

This is related to #19
  • Loading branch information
victorges authored May 3, 2022
1 parent 6d4f262 commit 576d4fd
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion task/runner.go
Original file line number Diff line number Diff line change
Expand Up @@ -144,9 +144,12 @@ func (r *runner) handleTask(ctx context.Context, taskInfo data.TaskInfo) (output

handler, ok := r.TaskHandlers[strings.ToLower(taskType)]
if !ok {
return nil, UnretriableError{fmt.Errorf("unknown task type=%q id=%s", taskType, taskID)}
return nil, UnretriableError{fmt.Errorf("unknown task type=%q", taskType)}
}

if taskCtx.Status.Phase == "running" {
return nil, UnretriableError{errors.New("task has already been started before")}
}
err = r.lapi.UpdateTaskStatus(taskID, "running", 0)
if err != nil {
glog.Errorf("Error updating task progress type=%q id=%s err=%q unretriable=%v", taskType, taskID, err, IsUnretriable(err))
Expand Down

0 comments on commit 576d4fd

Please sign in to comment.