Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement matrix sharding #492

Merged
merged 1 commit into from
Jan 10, 2025
Merged

Implement matrix sharding #492

merged 1 commit into from
Jan 10, 2025

Conversation

zanieb
Copy link
Member

@zanieb zanieb commented Jan 10, 2025

Since we've encountered the GitHub matrix size limit of 256 entries a couple times now (recently in #388), this adds sharding to our matrix generation script.

There's now a --max-shards option which splits the matrix into (up to) the requested number of shards. We don't split it evenly, we eagerly fill each shard first. To consume the shards, we just copy the build job into two identical child jobs which consume each matrix shard. Ideally, we'd avoid code duplication here by using a re-usable action or something for the build-steps — but that's overkill at this point.

We only need this for Linux for the foreseeable future.

@zanieb zanieb marked this pull request as draft January 10, 2025 19:52
@zanieb zanieb force-pushed the zb/shard-ci branch 10 times, most recently from a69bb3d to 121ac9e Compare January 10, 2025 21:43
@zanieb zanieb marked this pull request as ready for review January 10, 2025 21:47
@zanieb
Copy link
Member Author

zanieb commented Jan 10, 2025

e.g., new warning and error

❯ uv run -q ci-matrix.py > matrix.json
warning: matrix of size 288 exceeds limit of 256 but sharding is not enabled; consider setting `--max-shards`
❯ uv run -q ci-matrix.py --max-shards=1 > matrix.json
error: matrix of size 288 requires 2 shards, but the maximum is 1; consider increasing `--max-shards`

@zanieb
Copy link
Member Author

zanieb commented Jan 10, 2025

e.g., we create empty shards when empty

❯ uv run -q ci-matrix.py --labels arch:foo --platform darwin --max-shards=2 | jq
{
  "0": {
    "include": []
  },
  "1": {
    "include": []
  }
}

@zanieb zanieb merged commit cda1c64 into main Jan 10, 2025
300 checks passed
@zanieb zanieb deleted the zb/shard-ci branch January 10, 2025 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant