build(deps): upgrade actions/{upload,download}-artifact@v3 to v4

[Michael-J-Ward]

Note

The text of the PR has undergone significant changes since this morning on 2024-08-22. I noticed and tracked down some bugs in the build behavior that existed prior to this PR but went unnoticed. The result of that investigation is the two follow-on issues at the end.

Which issue does this PR close?

Closes #724.

Rationale for this change

v3 will be disabled 2024-11-30.

The change between actions/upload-artifact@v3 and v4 that impacted us was that each artifact is now immutable.

With v3, a wheel would be built for each target platform and then uploaded to the same dist artifact, which would aggregate the artifacts in an archive.

With v4, all but the 1st attempt to upload to dist fails with this error

Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

What changes are included in this PR?

To achieve this same behavior with v4, each build target must upload to its own unique key before merging them as a final step, as suggested in the migration guide.

Although the build workflow now produces a dist-* artifact for each build target, the final merged dist.zip artifact has the same structure as before.

Warning

Notice build.yml workflow should create 6 artifacts, but both dist.zip's only contain 4 artifacts. See Additional Context at end.

Build Summary from 40.1.0 release

❯ unzip -- -l dist.zip
Archive:  dist.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
 15145056  08-16-2024 01:38   datafusion-40.1.0-cp38-abi3-macosx_11_0_arm64.whl
 18462692  08-16-2024 01:41   datafusion-40.1.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 17550315  08-16-2024 01:41   datafusion-40.1.0-cp38-abi3-manylinux_2_28_aarch64.whl
 17850348  08-16-2024 01:48   datafusion-40.1.0-cp38-abi3-win_amd64.whl
---------                     -------
 69008411                     4 files

Build Summary from this PR

❯ unzip -- -l dist.zip
Archive:  dist.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
 15158055  08-22-2024 04:21   datafusion-40.0.0-cp38-abi3-macosx_11_0_arm64.whl
 18469550  08-22-2024 04:21   datafusion-40.0.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
 17566025  08-22-2024 04:21   datafusion-40.0.0-cp38-abi3-manylinux_2_28_aarch64.whl
 17862321  08-22-2024 04:21   datafusion-40.0.0-cp38-abi3-win_amd64.whl
---------                     -------
 69055951                     4 files

Are there any user-facing changes?

There shouldn't be.

Additional Context

This PR upgrades the action while maintaining current behavior. I assume we'll want a follow-on PR to address the two following warnings.

• update GH Build workflow now that macos-latest is ARM64 #831
• Update build-sdist job #832