package_test.go - Support OCI Image Layout #37727

andrewkroh · 2024-01-24T16:58:27Z

Proposed commit message

Modify TestDocker such that it can read both the original docker image layout and the OCI Image Layout. This works by reading the config and layer file names from the manifest.yml instead of assuming their names.

Fixes #37726

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Need to run it on a machine with an older docker and a new docker.

PLATFORMS=linux/amd64 SNAPSHOT=true mage -v package

Related issues

Fixes Flaky Test - TestDocker (a packaging test) fails validating exported images #37726

Notes

This is how the manifest.json looks under a docker export from Docker Engine 20.10.22.

[
  {
    "Config": "9c3a1c54bb292142bd87ab9710574289daf66725b924d6d2d7d2522abfa6c5e8.json",
    "RepoTags": [
      "docker.elastic.co/beats/auditbeat-oss:8.13.0-SNAPSHOT"
    ],
    "Layers": [
      "3889013043c4cd6303c1845ca9fe8ae182b4b4893f0a91f2830b9d42f8a84e3e/layer.tar",
      "17c0b5e4c288157c300ed8179ee331b164c8a8fe05a0122bdaab2bb74d36daaa/layer.tar",
      "886e9544ed7fc95e8ce46a620ba86541ce7559c38fa6600de6fd6bc2a1cabaaf/layer.tar",
      "eac6da820a3347f34d17c82183b41db94d46cae22022ca1a5c8a16cf7bd18091/layer.tar",
      "fd0873a272991097037f3c44b1a71f02cd243484eb65103372def58924604837/layer.tar",
      "a1e3c7b8704488fb882049f29d1208bbc8543094f605db639fbaa53e32747f48/layer.tar",
      "154241032891186627ef580ab1a6c1c9844a47209d96757cb079a329ee5fd46d/layer.tar",
      "fd4811cb85b6e681fcf4b72efaff36c25cd31e94fbef45623adc6627b662ccac/layer.tar",
      "5f6da050cc94f30a92ec75cd0cadcd9e8768c99435bc4a53f7765482ec1b06eb/layer.tar",
      "11ecf32a9786eba93435dc21a87a614e1449caba5395eb146245db32d82ea72e/layer.tar",
      "ada1b4fc98b714d383818dc39ccb40236cd2faacb4fb0d8cdc47e263cfb8e4be/layer.tar",
      "4273aa067d2f453ad799b011bd9d640fc4998cbd40b361d86f27901f7a639c0d/layer.tar"
    ]
  }
]

And this is how the manifest.json looks under a docker export from Docker Engine 25.0.1.

[
  {
    "Config": "blobs/sha256/75771bde63fc0e750f060facf4f8e983432438ea10a519539835f390509396a4",
    "RepoTags": [
      "auditbeat:latest"
    ],
    "Layers": [
      "blobs/sha256/b1099d79a48ef6b9eaf12e0083d6510f9d9a5f9085465344247ca2822a9687f0",
      "blobs/sha256/49386a6ffd28a05d5f0a84b9949a6a83cd4ce57c23a2bb11bb4ae37a42215db8",
      "blobs/sha256/0dfede3bba776760119b2987a1f8a472095da848fb9b200fec910a52aaa0c4b0",
      "blobs/sha256/664c054c208307b301a1cc4754b4a9ae0e76c71f2cab90158903d45687389011",
      "blobs/sha256/aa147e557a1f602b1477ea70b6a171104fd2e84be7d4e4c8deb3d5c59d196c07",
      "blobs/sha256/1600e739679c5cbfa4fd03f084c5ba9bd1fa8fbd977535d091b5c0a59607aa33",
      "blobs/sha256/464f69e40f0fe21164fd63a96a511ab749f61d25e36cb21f2b42479d3eeba844",
      "blobs/sha256/22388ba12e4c3cbd5ebde20992198f65b594112cb8e5b6d23c4fe2bd65d53711",
      "blobs/sha256/ebb14e3380c6f51e9928594c786cc47d317ccbfe731af24d95a6c1b9d67be8dd",
      "blobs/sha256/781df5ce7323957257aa66862883bdd2e9c578a899bca601bb411a1fe468089f",
      "blobs/sha256/66157a475ac27298e4b982627556968648c7b8074a500f619c5a747d3a12cc13",
      "blobs/sha256/5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
    ],
    "LayerSources": {
      "sha256:0dfede3bba776760119b2987a1f8a472095da848fb9b200fec910a52aaa0c4b0": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 26624,
        "digest": "sha256:0dfede3bba776760119b2987a1f8a472095da848fb9b200fec910a52aaa0c4b0"
      },
      "sha256:1600e739679c5cbfa4fd03f084c5ba9bd1fa8fbd977535d091b5c0a59607aa33": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 103729152,
        "digest": "sha256:1600e739679c5cbfa4fd03f084c5ba9bd1fa8fbd977535d091b5c0a59607aa33"
      },
      "sha256:22388ba12e4c3cbd5ebde20992198f65b594112cb8e5b6d23c4fe2bd65d53711": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 13824,
        "digest": "sha256:22388ba12e4c3cbd5ebde20992198f65b594112cb8e5b6d23c4fe2bd65d53711"
      },
      "sha256:464f69e40f0fe21164fd63a96a511ab749f61d25e36cb21f2b42479d3eeba844": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 1536,
        "digest": "sha256:464f69e40f0fe21164fd63a96a511ab749f61d25e36cb21f2b42479d3eeba844"
      },
      "sha256:49386a6ffd28a05d5f0a84b9949a6a83cd4ce57c23a2bb11bb4ae37a42215db8": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 51286016,
        "digest": "sha256:49386a6ffd28a05d5f0a84b9949a6a83cd4ce57c23a2bb11bb4ae37a42215db8"
      },
      "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 1024,
        "digest": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
      },
      "sha256:66157a475ac27298e4b982627556968648c7b8074a500f619c5a747d3a12cc13": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 345600,
        "digest": "sha256:66157a475ac27298e4b982627556968648c7b8074a500f619c5a747d3a12cc13"
      },
      "sha256:664c054c208307b301a1cc4754b4a9ae0e76c71f2cab90158903d45687389011": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 4096,
        "digest": "sha256:664c054c208307b301a1cc4754b4a9ae0e76c71f2cab90158903d45687389011"
      },
      "sha256:781df5ce7323957257aa66862883bdd2e9c578a899bca601bb411a1fe468089f": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 6144,
        "digest": "sha256:781df5ce7323957257aa66862883bdd2e9c578a899bca601bb411a1fe468089f"
      },
      "sha256:aa147e557a1f602b1477ea70b6a171104fd2e84be7d4e4c8deb3d5c59d196c07": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 4096,
        "digest": "sha256:aa147e557a1f602b1477ea70b6a171104fd2e84be7d4e4c8deb3d5c59d196c07"
      },
      "sha256:b1099d79a48ef6b9eaf12e0083d6510f9d9a5f9085465344247ca2822a9687f0": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 68108288,
        "digest": "sha256:b1099d79a48ef6b9eaf12e0083d6510f9d9a5f9085465344247ca2822a9687f0"
      },
      "sha256:ebb14e3380c6f51e9928594c786cc47d317ccbfe731af24d95a6c1b9d67be8dd": {
        "mediaType": "application/vnd.oci.image.layer.v1.tar",
        "size": 3010560,
        "digest": "sha256:ebb14e3380c6f51e9928594c786cc47d317ccbfe731af24d95a6c1b9d67be8dd"
      }
    }
  }
]

mergify · 2024-01-24T16:59:11Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @andrewkroh? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

Modify TestDocker such that is can read both the original docker image layout and the OCI Image Layout. This works by reading the config and layer file names from the manifest.yml instead of assuming their names. Fixes elastic#37726

cmacknz

Thanks for this!

cmacknz · 2024-01-24T17:28:32Z

dev-tools/packaging/package_test.go

 			info, err = readDockerInfo(tarReader)
 			if err != nil {
 				return nil, nil, err
 			}
-		case strings.HasSuffix(header.Name, "/layer.tar"):
+		case slices.Contains(manifest.Layers, header.Name):


This isn't in 8.12 yet since it is still at Go 1.20, so is 7.17 but we plan to change that with #37694 since 7.17.18 will be after the Go 1.22 release.

I don't think we can put it into 8.12 since Go 1.21 modifies our support matrix by dropping Windows 8.1

When we do the backport for 8.12 we can substitute in golang.org/x/exp/slices or a helper function or a map.

elasticmachine · 2024-01-24T19:15:19Z

❕ Build Aborted

Either there was a build timeout or someone aborted the build.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 136 min 20 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-24T20:00:01Z

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2024-01-24T17:16:23.244+0000
Duration: 163 min 23 sec

Test stats 🧪

Test	Results
Failed	0
Passed	28782
Skipped	2015
Total	30797

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

elasticmachine · 2024-01-24T20:32:23Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Duration: 190 min 14 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Modify TestDocker such that is can read both the original docker image layout and the OCI Image Layout. This works by reading the config and layer file names from the manifest.yml instead of assuming their names. Fixes #37726 (cherry picked from commit ebd8512)

Modify TestDocker such that is can read both the original docker image layout and the OCI Image Layout. This works by reading the config and layer file names from the manifest.yml instead of assuming their names. Fixes #37726 (cherry picked from commit ebd8512) Co-authored-by: Andrew Kroh <[email protected]>

…37730) * package_test.go - Support OCI Image Layout (#37727) Modify TestDocker such that is can read both the original docker image layout and the OCI Image Layout. This works by reading the config and layer file names from the manifest.yml instead of assuming their names. Fixes #37726 (cherry picked from commit ebd8512) * Replace slices.Contains usage with helper func The 8.12 branch is not yet on Go 1.21 so the slices package isn't available.

Modify TestDocker such that is can read both the original docker image layout and the OCI Image Layout. This works by reading the config and layer file names from the manifest.yml instead of assuming their names. Fixes elastic#37726

andrewkroh added bug enhancement labels Jan 24, 2024

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 24, 2024

andrewkroh added :Packaging Packaging Team:Elastic-Agent Label for the Agent team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jan 24, 2024

mergify bot assigned andrewkroh Jan 24, 2024

andrewkroh mentioned this pull request Jan 24, 2024

Flaky Test - TestDocker (a packaging test) fails validating exported images #37726

Closed

package_test.go - Support OCI Image Layout

66bffc7

Modify TestDocker such that is can read both the original docker image layout and the OCI Image Layout. This works by reading the config and layer file names from the manifest.yml instead of assuming their names. Fixes elastic#37726

andrewkroh force-pushed the bugfix/package/docker-oci-image-layout branch from e66ddb2 to 66bffc7 Compare January 24, 2024 17:16

andrewkroh marked this pull request as ready for review January 24, 2024 17:20

andrewkroh requested a review from a team as a code owner January 24, 2024 17:20

andrewkroh requested review from rdner and faec January 24, 2024 17:20

Add missing file Close()

17f2618

cmacknz added backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.12.0 Automated backport with mergify labels Jan 24, 2024

cmacknz approved these changes Jan 24, 2024

View reviewed changes

cmacknz mentioned this pull request Jan 24, 2024

Update to Go 1.21.6 #37694

Merged

cmacknz reviewed Jan 24, 2024

View reviewed changes

andrewkroh merged commit ebd8512 into elastic:main Jan 24, 2024
122 checks passed

mergify bot mentioned this pull request Jan 24, 2024

[7.17](backport #37727) package_test.go - Support OCI Image Layout #37729

Merged

mergify bot mentioned this pull request Jan 24, 2024

[8.12](backport #37727) package_test.go - Support OCI Image Layout #37730

Merged

michalpristas mentioned this pull request Jan 25, 2024

package_test.go - Support OCI Image Layout elastic/elastic-agent#4139

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

package_test.go - Support OCI Image Layout #37727

package_test.go - Support OCI Image Layout #37727

andrewkroh commented Jan 24, 2024 •

edited

Loading

mergify bot commented Jan 24, 2024

cmacknz left a comment

cmacknz Jan 24, 2024

andrewkroh Jan 24, 2024

elasticmachine commented Jan 24, 2024

Build stats

elasticmachine commented Jan 24, 2024

Build stats

Test stats 🧪

elasticmachine commented Jan 24, 2024

Build stats

package_test.go - Support OCI Image Layout #37727

package_test.go - Support OCI Image Layout #37727

Conversation

andrewkroh commented Jan 24, 2024 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Related issues

Notes

mergify bot commented Jan 24, 2024

cmacknz left a comment

Choose a reason for hiding this comment

cmacknz Jan 24, 2024

Choose a reason for hiding this comment

andrewkroh Jan 24, 2024

Choose a reason for hiding this comment

elasticmachine commented Jan 24, 2024

❕ Build Aborted

Build stats

🤖 GitHub comments

elasticmachine commented Jan 24, 2024

❕ Build Aborted

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Jan 24, 2024

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

andrewkroh commented Jan 24, 2024 •

edited

Loading