GH-43951: [CI][Python] Use GitHub Packages for vcpkg cache #44644
Conversation
|
|
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
|
kou
force-pushed
the
ci-wheel-manylinux-vcpkg-cache
branch
from
c931193
to
f401684
Compare
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
1 similar comment
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
kou
changed the title
kou
force-pushed
the
ci-wheel-manylinux-vcpkg-cache
branch
from
78ee981
to
8b4f8ab
Compare
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
2 similar comments
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
kou
changed the title
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
1 similar comment
|
@github-actions crossbow submit wheel-manylinux-2-28-cp39-cp39-amd64 |
This comment was marked as outdated.
This comment was marked as outdated.
kou
force-pushed
the
ci-wheel-manylinux-vcpkg-cache
branch
from
acf6b87
to
c75a117
Compare
|
@github-actions crossbow submit java-jars wheel-manylinux-2014-cp39-cp39-* |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit java-jars |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit -g wheel java-jars |
|
Revision: 63d255c Submitted crossbow builds: ursacomputing/crossbow @ actions-e75f942933
|
There was a problem hiding this comment.
This is ready.
manylinux and java-jar jobs use NuGet + GitHub Packages based cache.
Exception: manylinux2014 + aarch64 jobs don't use it. Because NuGet doesn't work on the platform. (Mono is old.)
We can't use GitHub Actions cache because Crossbow uses only feature (?) branches. We can't use GitHub Actions cache efficiently without the default branch integration.
| GITHUB_TOKEN="${GITHUB_TOKEN}" \ | ||
| VCPKG_BINARY_SOURCES="clear;nuget,GitHub,readwrite" \ | ||
| VCPKG_ROOT=C:\\vcpkg | ||
| # TODO: Use --mount=type=secret for GITHUB_TOKEN |
There was a problem hiding this comment.
I'll work on this in a follow-up task.
github-actions
bot
added
awaiting changes
|
|
||
| # TODO: Use --mount=type=secret for GITHUB_TOKEN |
There was a problem hiding this comment.
Is there an issue open for this? If so, can we mention the GH issue number?
There was a problem hiding this comment.
No. I'll open it when this is merged or someone agrees that we should do it.
| VCPKG_FORCE_SYSTEM_BINARIES=1 \ | ||
| VCPKG_OVERLAY_TRIPLETS=/arrow/ci/vcpkg \ | ||
| GITHUB_REPOSITORY_OWNER="${GITHUB_REPOSITORY_OWNER}" \ | ||
| GITHUB_TOKEN="${GITHUB_TOKEN}" \ |
There was a problem hiding this comment.
I don't think we should merge it like this, it bakes the token into the image. Even if the token is ephemeral it could still be abused (it lives a little longer than the job) and crossbow jobs have full permissions because they don't run as external pull requests.
In addition if people test/build the image locally it could leak their token as well.
Rationale for this change
We're using only Docker level cache for vcpkg used for wheels. If we have any vcpkg related changes, all vcpkg ports are rebuilt. It's time consuming.
What changes are included in this PR?
Enable NuGet + GitHub Packages based cache. It's port level cache. So we don't need to rebuild all ports when we have any vcpkg related changes.
See also: https://learn.microsoft.com/en-us/vcpkg/consume/binary-caching-github-packages
NuGet + GitHub Packages based cache isn't enabled with manylinux2014 + aarch64. Because EPEL for CentOS 7 + aarch64 provides old Mono. (FYI: EPEL for CentOS 7 + x86_64 provides newer Mono.) We can't use old Mono to run NuGet on Linux.
Are these changes tested?
Yes.
Are there any user-facing changes?
No.