bundler: Support prefetching binary dependencies

[a-ovchinnikov]

Some dependencies contain architecture-specific components. Their name and location differ from standard Ruby-only dependencies. As a result prior to this commit such dependencies would be skipped during prefetch. This, in turn, failed hermetic builds because a dependency would end up missing.

Resolves: #672

Maintainers will complete the following section

• Commit messages are descriptive enough
• Code coverage from testing does not decrease and new code is covered
• Docs updated (if applicable)
• Docs links in the code are still valid (if docs were updated)

Note: if the contribution is external (not from an organization member), the CI
pipeline will not run automatically. After verifying that the CI is safe to run:

• approve GitHub Actions workflows by clicking a button
• approve the Red Hat Trusted App Pipeline container build by commenting /ok-to-test
  (as is the standard for Pipelines as Code)

[a-ovchinnikov]

/retest

[brunoapimentel]

/retest

[slimreaper35]

It would be fine to keep e2e tests as separate commits. You forgot to generate test data.

[slimreaper35]

I am not sure if overall that's what we want. When I was testing it locally, it looked like our parser script always adds platform = "ruby" if the source gem exists. So if users set allow binary to true, we download source gems anyway.

When I had (sorbet)[https://rubygems.org/gems/sorbet-static/versions] in Gemfile.lock. JSON output had 3 gems for 3 different platforms. Then, we would not download all pre-compiled gems available as we do in pip.

Everything kind of depends on the parser.

[a-ovchinnikov]

I am not sure if overall that's what we want.

My understanding is that we want to download whatever is specified in a Gemfile.lock. Technically we do not even need to ask a user if they want precompiled binaries or not, but for consistency with pip we do this. I think it is also good to be explicit about binary blobs and make user acknowledge that this is indeed what they want: there is always a chance that this was never noticed before and made its way into a stricter environment by accident.

[eskultety]

Mostly minor comments on top of @slimreaper35's . The default bundler version downgrade behaviour raised a red flag for me and so I believe we need to set BUNDLE_VERSION.

[slimreaper35]

I miss generated SBOM from e2e tests.

[slimreaper35]

One more thing, that I probably mentioned somewhere, if we want to add platform qualifier in the PURLs:
https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#gem

[eskultety]

One more thing, that I probably mentioned somewhere, if we want to add platform qualifier in the PURLs: https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#gem

Good point.

[a-ovchinnikov]

One more thing, that I probably mentioned somewhere, if we want to add platform qualifier in the PURLs: https://github.com/package-url/purl-spec/blob/master/PURL-TYPES.rst#gem

Good point.

If I read the link correctly this is a different platform: not a platform a gem is compiled to, but a Ruby implementation. The given example explicitly mentions JRuby.

[slimreaper35]

If I read the link correctly this is a different platform: not a platform a gem is compiled to, but a Ruby implementation. The given example explicitly mentions JRuby.

Okay. So we can probably leave it. We are not handling this for pip either.
And how about the SBOM from e2e tests ?

[a-ovchinnikov]

And how about the SBOM from e2e tests ?

Do we need it? The logic of SBOM generation is pretty static and well-tested with UTs (97+% coverage is no joke). The tests check that bundler can build a package after it was prepared by cachi2 and that there are no failures there, they do catch quite a lot of interesting behaviours as we have learned. I think adding more data is excessive and will not bring any more confidence.

[slimreaper35]

We have it for all e2e tests for all package managers, so at least for consistency, I would add it. We can discuss it when we put all integration tests into a single repo. That should be a good starting point for improving the integration tests overall.

[a-ovchinnikov]

We have it for all e2e tests for all package managers

Done!