-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci: improve performance by eliminating redundancies #14028
Conversation
d50c96b
to
0dd009b
Compare
`make install` is being redundantly run three times for each E2E test: 1. In the workflow: https://github.com/argoproj/argo-workflows/blob/6699ab396f830210f6dcac4f00a9328a629c142f/.github/workflows/ci-build.yaml#L349-L350 2. As a prerequisite of `make start`: https://github.com/argoproj/argo-workflows/blob/6699ab396f830210f6dcac4f00a9328a629c142f/Makefile#L547o 3. As a dependency by Kit: https://github.com/argoproj/argo-workflows/blob/6699ab396f830210f6dcac4f00a9328a629c142f/tasks.yaml#L39 Similarly, the changes in argoproj#14012 to centralize binary building aren't effective because the binaries are being rebuilt. This eliminates the redundancy by removing the unnecessary prerequisites from the `Makefile` and the redundant `make install` step in `ci-build.yaml`. Also, I added `make --touch dist/*` to mark everything as up-to-date so the binaries aren't rebuilts (docs: https://www.gnu.org/software/make/manual/html_node/Instead-of-Execution.html) Signed-off-by: Mason Malone <[email protected]>
0dd009b
to
c962c0f
Compare
Signed-off-by: Mason Malone <[email protected]>
Signed-off-by: Mason Malone <[email protected]>
/retest |
👍Would you have any metrics on how much performance this improves? |
@tczhao The first two redundancies impact the time for the "Start controller/API" step in the E2E test jobs. The new "Performance Metrics" page can show metrics for the E2E jobs as a whole, but it doesn't look at has step-level granularity. From a very quick look, it appears it reduces the time for that step by 2-3 minutes. The third redundancy impacts everything Go-related, and the fix in this PR to avoid re-downloading Go dependencies isn't going to help with that until the cache is updated. I think it'll be 10-20 seconds. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
Motivation
There's three sources of redundancy that are slowing down CI builds:
make install
is being redundantly run three times for each E2E test:argo-workflows/.github/workflows/ci-build.yaml
Lines 349 to 350 in 6699ab3
make start
:argo-workflows/Makefile
Line 547 in 6699ab3
argo-workflows/tasks.yaml
Line 39 in 6699ab3
controller
andcli
are prerequisites ofmake start
, and listed as prerequisites in Kit.GOMODCACHE
. I think the problem is the cache is usually populated by thedocs
workflow, which doesn't fetch all the Go dependencies (example), so they aren't included whenactions/setup-go
saves the cache.Modifications
This eliminates the first two redundancies by removing the unnecessary prerequisites from the
Makefile
and the redundantmake install
step inci-build.yaml
. Also, I addedmake --touch dist/*
to mark everything as up-to-date so the binaries aren't rebuilt (docs). For the issue with the go dependencies not being in the cache, I updateddocs.yaml
to rungo mod download
to ensure they're included.Also, I consolidated
Start controller/API
andWait for controller to be up
steps so you can see everything Kit is doing in the logs for the former, which will help prevent these kind of oversights in the future. The reason I didn't catch the issue with binaries being rebuilt in #14012 is because the logs are discarded unless the build failed (in which case they were visible in theFailure debug - Controller/API logs
step).Verification
Wait for E2E tests