Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#12840: Add more handling more multiple attempts by restricting the space of github_job_ids we're looking to only the ones in the workflow run attempt in question #12858

Merged
merged 2 commits into from
Sep 18, 2024

Conversation

tt-rkim
Copy link
Collaborator

@tt-rkim tt-rkim commented Sep 18, 2024

Ticket

#12840

Problem description

We weren't handling the case properly where artifacts would carry over from successful jobs from previous workflow run attempts. However, at the same time, based on anecdotal sniffing of the GitHub API, it seems that jobs in subsequent workflow run attempts will actually have different IDs.

This means we got into a situation where in our uuid -> github_job_id mapping, we could have multiple github_job_ids for the same uuid. This caused a keyerror in certain cases.

What's changed

  • We restrict our view of GitHub job IDs to only the ones which are relevant
  • (more importantly + correctly) We go through all available log files for each github_job_id and construct a mapping between github_job_id -> uuid rather than other way around. This causes clashes for a given uuid.

Checklist

  • Post commit CI passes
  • Blackhole Post commit (if applicable)
  • Model regression CI testing passes (if applicable)
  • Device performance regression CI testing passes (if applicable)
  • New/Existing tests provide coverage for changes

…g all of them, so that the system doesn't get confused when searching for artifacts that were generated in a previous attempt but weren't carried over
… downloaded logs to a test report, only match the ones for the requested job, so we don't handle more than we need to and run into issues such as what we currently have where multiple attempts of a workflow run will cause multiple github_job_ids to have the same uuid, meaning we can't match later because we lose data
@tt-rkim tt-rkim merged commit ea8522c into main Sep 18, 2024
6 checks passed
@tt-rkim tt-rkim deleted the rkim/12840-multiple-attempts branch September 18, 2024 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant