CI: enabling easy caching in gitlab CI.

[VannTen]

What would you like to be added

the cache keyword in gitlab CI don't work out of the box, we're hitting warning like:

Checking cache for pre-commit-all-3-non_protected...
No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. 
WARNING: Cache file does not exist                 
Failed to extract cache

We should set-up the different runners so that it works out of the box

Why is this needed

It would improve performance of the CI significantly (IMO) and avoid some problems like rate-limiting we face regularly (see #11485 )

/cc @ant31

[ant31]

For shared cache we would need something like S3.
The alternative is to use a local storage. It can work fine because we're using a limited number of fixed nodes.

Configuration is here:
https://github.com/kubespray/kspray-infra/blob/main/apps/base/cicd/gitlab-ci/gitlab-runner/manifests/configs/config.template.toml
Maintainers can do a PR; once merged, it's deployed to the CI cluster.

[VannTen]

I'll take a look 👍

[VannTen]

@ant31 From the prometheus it seems there is about 2TB of storage on each node, is that right ?
I guess we could put a minio using the local-provisioner and use that as an S3 cache ? I don't know much about minio resources consumption, would that take much capacity from the CI itself ?

[ant31]

minio is deployed and ci uses cache in: #11671
The boxes are not fetched from internet anymore if the cache exists.

Cache can be used on all jobs (e.g pip install etc), to be configured overtime