No concurrent builds of multi arch CLI runs #1000
Assignees
Labels
Comments
jakobmoellerdev
added a commit
that referenced
this issue
Nov 4, 2024
…ncurrency (#996) Our caches are getting too big: ``` Approaching total cache storage limit (58.09 GB of 10 GB Used) ``` On top we dont make use of makes inbuilt parallelism (`-j`) and also we do not make use of shells forking mechanism for concurrent execution (`&` execution chain followed by `wait`) <!-- markdownlint-disable MD041 --> #### What this PR does / why we need it Limits the cache save to only the main branch in the build step, all other branches/PRs and Jobs will reuse that cache instead of computing their own What does this mean: PRO: - only one cache that should roundabout have 4 (build files for all platforms) + 2 (build+test files for unit tests on amd64) Gi of modules and builds that get updated on every commit to main - The build cache is likely unaffected or largely still valid for most if not every PR. This leads to compile time improvements upwards of 600% depending on PR size without killing our cache usage. CON: - a rebuild or dependency changes will be able to "invalidate" that cache and could lead to longer build times faster. However in general, most builds should still perform the same. If the go.mod / go.sum is not touched, the build cache will be restored for all files except the ones touched or impacted by the PR. #### Which issue(s) this PR fixes <!-- Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`. --> fix #999 #1000 --------- Co-authored-by: Hilmar Falkenberg <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
Whenever we build the CLI, we build our architectures one by one. Because they dont share a build cache for large sections, we should concurrently run these compilations and let the bottleneck be the runner that we compile on.
What you expected to happen:
I expect all make steps for compilation to run with
make -jfor parallelization through make, and all shell scripts with a for loop for compilation through archs to run via a forked process through&andwaitchainingHow to reproduce it (as minimally and precisely as possible):
Call
make buildormake ctfon any componentAnything else we need to know:
Environment:
The text was updated successfully, but these errors were encountered: