Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add cron job to cache trivy db #1918

Merged
merged 1 commit into from
Nov 8, 2024

Conversation

binbin-li
Copy link
Collaborator

@binbin-li binbin-li commented Nov 6, 2024

Description

What this PR does / why we need it:

The trivy DB throttling is getting worse recently. We added retry logic before to mitigate the issue, now we need to cache the DB daily to reduce requests to the trivy DB.

  1. Add a new workflow to cache trivy DB dailiy. The oras pull command would be retried 10 times with 1 minute interval if it fails.
  2. Restore the trivy DB cache for each job that requires trivy scan.
  3. Remove retry logic in Makefile.

I've run tests in my forked repo, check status:
https://github.com/binbin-li/ratify/actions/runs/11705539729
https://github.com/binbin-li/ratify/actions/runs/11705756491
https://github.com/binbin-li/ratify/actions/runs/11705780438

And the update DB job can be only run on dev branch.
image

Which issue(s) this PR fixes (optional, using fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when the PR gets merged):

Fixes #

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Helm Chart Change (any edit/addition/update that is necessary for changes merged to the main branch)
  • This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Please also list any relevant details for your test configuration

  • Test A
  • Test B

Checklist:

  • Does the affected code have corresponding tests?
  • Are the changes documented, not just with inline documentation, but also with conceptual documentation such as an overview of a new feature, or task-based documentation like a tutorial? Consider if this change should be announced on your project blog.
  • Does this introduce breaking changes that would require an announcement or bumping the major version?
  • Do all new files have appropriate license header?

Post Merge Requirements

  • MAINTAINERS: manually trigger the "Publish Package" workflow after merging any PR that indicates Helm Chart Change

Copy link

codecov bot commented Nov 6, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

see 133 files with indirect coverage changes

@binbin-li binbin-li force-pushed the cache-trivy-db branch 3 times, most recently from da3a91c to 288f363 Compare November 7, 2024 03:28
@akashsinghal
Copy link
Collaborator

Looks like this PR is running into failures that would happen if the daily job failed to populate the cache :). I think we can override checks to get this in

@binbin-li binbin-li merged commit 4ab047b into ratify-project:dev Nov 8, 2024
17 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants