From 00299084978648c694603628d4d96b3fa266d35e Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Tue, 29 Aug 2023 11:37:13 -0700 Subject: [PATCH 1/9] Add draft for bundler checksum verification --- text/0011-gem-checksum-verification.md | 144 +++++++++++++++++++++++++ 1 file changed, 144 insertions(+) create mode 100755 text/0011-gem-checksum-verification.md diff --git a/text/0011-gem-checksum-verification.md b/text/0011-gem-checksum-verification.md new file mode 100755 index 0000000..3058050 --- /dev/null +++ b/text/0011-gem-checksum-verification.md @@ -0,0 +1,144 @@ +- Feature Name: (fill me in with a unique ident, my\_awesome\_feature) +- Start Date: (fill me in with today's date, YYYY-MM-DD) +- RFC PR: (leave this empty) +- Bundler Issue: (leave this empty) + +# Summary + +Bundler is adding checksum verification of gems when they are installed. It should be secure by default and easy to use. It should not break assumptions or unnecessarily break deployment or CI unless there is a security problem. + +# Motivation + +Verifying gem checksums with known checksums at install time is a stronger way to verify that the exact same gem source is being used in every environment where the bundle is installed. Using checksums, especially those sourced from rubygems.org, is a more strict way of locking to a specific gem version, and the feature should work mostly transparently in the same way that specific versions are locked currently. + +# Guide-level explanation + +Upon first upgrading to bundler 2.5.0, the checksum feature will be enabled. + +First usage will fall into two use cases: +1. I want to immediately use the new checksums feature. +2. I’m not specifically using the checksums feature, but expect bundler to be secure automatically. + +For users excited about the new feature, the usual commands and docs should show the new options for interacting with checksums. Docs and the release notice should include the most direct way to fully use the checksums. + +For users that are specifically looking for the feature, common commands like `bundle install`, `bundle update`, `bundle lock`, `bundle add`, should automatically make use of checksums when available. When there are missing checksums that must be fetched with a specific interaction, the UI should say what that action is. + +Example: + +``` +$ bundle install +Bundle complete! 88 Gemfile dependencies, 256 gems now installed. +Use `bundle info [gemname]` to see where a bundled gem is installed. +256 gems are missing checksums. Source integrity could not be verified. +Use `bundle lock` to add checksums to Gemfile.lock. +``` + +When a gem version is updated in the lockfile, the checksum should be recorded. Running `bundle update` or `bundle add` should record the checksum from the source into the Gemfile.lock. If a checksum is not available from the source because the source does not provide such info, then a checksum should be created during install, or a warning should be printed. + +If a user wishes to ensure that the environment they are using contains only gems installed from verified sources that match the checksums in the Gemfile.lock, the command `bundle pristine` should be run. The `bundle pristine` command installs every gem fresh from the .gem source, but does not fetch gems from the server or read the source index. `pristine` should be performed after checksums are written using the above commands. + +Example: + +``` +$ bundle pristine +Installing rake 13.0.6 +… +Installing rspec 3.12.0 +42 gems are missing checksums. Source integrity could not be verified. +Use `bundle lock` to add checksums to Gemfile.lock. +$ bundle lock --update +Writing lockfile to path/to/Gemfile.lock +$ bundle pristine +Installing rake 13.0.6 (verified) +… +Installing rspec 3.12.0 (verified) +$ +``` + +If a gem that is about to be installed does not match the checksum, an error is generated. + +Example: + +``` +$ bundle install +Installing rake 13.0.6 +Bundler cannot continue installing rake-13.0.6. +The checksum for the downloaded `rake-13.0.6.gem` does not match the known checksum for the gem. + +Expected: sha256-814828c34f1315d7e7b7e8295184577cc4e969bad6156ac069d02d63f58d82e8 from: Gemfile.lock:21:1 CHECKSUMS rake (13.0.6) +Downloaded: sha256-69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b6 from: +path/to/rake-13.0.6.gem SHA256 hexdigest + +To resolve this issue: +Delete the downloaded gem files referenced above and run `bundle install` + +If you are sure that the downloaded gem is correct, you can run… +(RFC TODO: Currently undecided on what the command should be here to allow unverified gems.) +`bundle install --skip-verification` OR +`bundle config set unverified ignore` OR +Remove the offending line from CHECKSUMS in Gemfile.lock +``` + +Certain checksums will always be unavailable because the source does not provide a checksum. When a checksum is not available, a line can be added to the Gemfile.lock CHECKSUMS section that explicitly indicates that no checksum is recorded for that gem version. Future installs of gems with intentionally ignored checksums should not error or warn when the gem is installed. The (RFC TODO) command will allow writing ignored checksum versions. Alternatively, the SHA256 digest of the gem at install time can be recorded, which will allow the user to verify future installs do not deviate from the recorded checksum. + +Platforms may pose a problem for Gemfile.lock checksums. If a user has not added one of the platforms where they bundle their app, then checksum verification could be skipped. In order to avoid this security vulnerability, the following should happen automatically. + +1. When a gem is installed and no checksum is available in the lockfile, a warning should be printed indicating that unverified gems were installed. +2. When a gem is installed where a checksum is available for the same gem on a different platform, an error should be raised. (i.e. if nokogiri for darwin has a checksum, but nokogiri for musl does not, then an error should be raised). +3. When bundle install is configured with frozen, any gem installed that does not have a checksum (and it was not explicitly ignored) should cause an error. + +When platform or missing checksum issues are raised, the error should always indicate what command should be run to fix the error. + +Bundling the app in CI should highlight any platform issues if the CI environment matches the production environment. This will make the bundler upgrade progress more smoothly for users. + +To encourage uptake of this security feature, it should be enabled by default and any actions that have already fetched checksums should record them in the Gemfile.lock. Progressive upgrading to verified checksum behavior should happen naturally unless explicitly disabled. + +Gem upgrades in the Gemfile.lock happen in two ways currently. Each way will behave differently. + +1. The user explicitly runs `bundle update`. +2. Dependabot, or another automation, changes the version of a locked gem directly in the Gemfile.lock (assuming that dependabot is not upgraded to also change the checksum) + +Explicit gem updates will record the new checksum for the gem and remove any old checksums for gems that are no longer in the Gemfile.lock. + +Direct updates will print a warning during install that X number of gem(s) could not be verified. If the bundle is frozen, then the warning will instead print an error and halt gem installation. + +# Reference-level explanation + +This is the technical portion of the RFC. Explain the design in sufficient detail that: + +- Its interaction with other features is clear. +- It is reasonably clear how the feature would be implemented. +- Corner cases are dissected by example. + +The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. + +# Drawbacks + + +Excessive failures + +If checksum verification failures happen more often than expected, it could cause the feature to be ignored or derided as poorly designed and implemented. Bundler should progressively update the Gemfile.lock transparently without too much interaction or excessive warnings and failures. Since checksums verification is just a more strict approach to version verification, this feature fits with the existing expectations of Bundler’s features. This means that checksum verification should be unobtrusive. The feature is not a big deal that needs lots of messaging, and should not fail unless bundler is configured to be strict (in the case of missing checksums) or there is an actual verification failure (in the case of corrupt or malicious gems) + +Increased installation time + +Gems will be SHA256 digested during install, adding to the install time. This could be a significant burden for some bundles. Future proposals such as content addressable storage for gem data could address this slowdown, but this is not immediately realistic. + +Unverifiable gems + +When gems come from private gem servers that do not implement the compact index, checksums will not be available. Bundler must provide configuration that can automatically store digests for gems from sources that don’t supply checksums so that the feature is not a burden to users relying on these sources. An option could be created to configure a specified source to always compute checksums for updated versions. This would mean any change in version from the source would be trusted at the time, but any locked version that is installed later would be verified to match the first install. + +GitHub Dependabot version upgrades are at risk of failing. + +If a user configures CI to use locked gem versions, it would be necessary for Dependabot to write the corresponding checksum line to the Gemfile to prevent every CI build from failing. This could cause many users to disable the feature because it interferes with an existing workflow or causes excessive CI failure notices. This can be addressed by making clear messaging about how to fix this problem. + +# Rationale and Alternatives + +- Rubygems.org already stores SHA256 checksums for gems and returns them in the compact index response. All the information is already present for checksum verification on the client side. +- Verifying .gem files at install time offers nearly complete protection against hacked, altered and corrupted gems. Bundler already ensures that only the bundled gems are available to the app. This feature adds the assurance that only verified gems were installed with the bundle. +- One alternative to this solution is including (vendoring) the bundled gems in the repository. This effectively has the same result, since the gems that will be installed during the production deploy will be verifiably the same gems that were used during CI and development. The downside of this vendored approach is the increase in the repository size. Checksums allow for a similar level of confidence without the downsides of vendoring gems and can be enabled by default so that more users will benefit. + + +# Unresolved questions + +- What are the exact commands, options, warning and error messages that are used to interact with the feature? +- What are the unexpected errors that may happen the first time people verify the gems they are downloading? Are there any errors or problems in the existing rubygems.org repository? From 3941ef0e638b6d4245aa8e63b6fbad3320340ae0 Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Tue, 29 Aug 2023 11:39:43 -0700 Subject: [PATCH 2/9] Add headings --- text/0011-gem-checksum-verification.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0011-gem-checksum-verification.md b/text/0011-gem-checksum-verification.md index 3058050..0b0e502 100755 --- a/text/0011-gem-checksum-verification.md +++ b/text/0011-gem-checksum-verification.md @@ -1,6 +1,6 @@ -- Feature Name: (fill me in with a unique ident, my\_awesome\_feature) -- Start Date: (fill me in with today's date, YYYY-MM-DD) -- RFC PR: (leave this empty) +- Feature Name: `gem_checksum_verification` +- Start Date: 2023-08-29 +- RFC PR: https://github.com/rubygems/rubygems/pull/6374 - Bundler Issue: (leave this empty) # Summary From 5958473b198a92352a79032a81cc99b9a338a0b1 Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Tue, 29 Aug 2023 13:13:54 -0700 Subject: [PATCH 3/9] Clarify language and add Reference-level explanation --- text/0011-gem-checksum-verification.md | 58 +++++++++++++++----------- 1 file changed, 33 insertions(+), 25 deletions(-) diff --git a/text/0011-gem-checksum-verification.md b/text/0011-gem-checksum-verification.md index 0b0e502..ecbc284 100755 --- a/text/0011-gem-checksum-verification.md +++ b/text/0011-gem-checksum-verification.md @@ -9,19 +9,19 @@ Bundler is adding checksum verification of gems when they are installed. It shou # Motivation -Verifying gem checksums with known checksums at install time is a stronger way to verify that the exact same gem source is being used in every environment where the bundle is installed. Using checksums, especially those sourced from rubygems.org, is a more strict way of locking to a specific gem version, and the feature should work mostly transparently in the same way that specific versions are locked currently. +Verifying gem checksums with known checksums at install time is a stronger way to verify that the exact same gem source is being used in every environment where the bundle is installed. Using checksums, especially those sourced from rubygems.org, is a more strict way of locking to a specific gem version. The feature should work transparently in much the same way that bundler already locks to specific versions. # Guide-level explanation -Upon first upgrading to bundler 2.5.0, the checksum feature will be enabled. +Upon first upgrading to the version of Bundler with this feature, the checksum feature will be enabled. First usage will fall into two use cases: -1. I want to immediately use the new checksums feature. -2. I’m not specifically using the checksums feature, but expect bundler to be secure automatically. +1. Users that want to immediately use the new checksums feature. +2. Users that are not specifically using the checksums feature, but expect bundler to work securely without interference. -For users excited about the new feature, the usual commands and docs should show the new options for interacting with checksums. Docs and the release notice should include the most direct way to fully use the checksums. +For users excited about the new feature, the usual commands and docs should show the new options for securing the bundle with checksums. Documentation and the release notice should include instructions for directly using the new feature. -For users that are specifically looking for the feature, common commands like `bundle install`, `bundle update`, `bundle lock`, `bundle add`, should automatically make use of checksums when available. When there are missing checksums that must be fetched with a specific interaction, the UI should say what that action is. +For users that are not specifically looking for the feature, common commands like `bundle install`, `bundle update`, `bundle lock`, `bundle add`, should automatically make use of checksums when available. When there are missing checksums that must be fetched with a specific interaction, the UI should say what that action is. Example: @@ -33,9 +33,9 @@ Use `bundle info [gemname]` to see where a bundled gem is installed. Use `bundle lock` to add checksums to Gemfile.lock. ``` -When a gem version is updated in the lockfile, the checksum should be recorded. Running `bundle update` or `bundle add` should record the checksum from the source into the Gemfile.lock. If a checksum is not available from the source because the source does not provide such info, then a checksum should be created during install, or a warning should be printed. +When a gem version is updated in the lockfile using a source that has checksums (like rubygems.org), the checksum should be recorded. Running `bundle update` or `bundle add` should record the checksum from the source into the Gemfile.lock. If a checksum is not available from the source because the source does not provide such info, then a checksum should be created during install using the .gem file or a warning should be printed. -If a user wishes to ensure that the environment they are using contains only gems installed from verified sources that match the checksums in the Gemfile.lock, the command `bundle pristine` should be run. The `bundle pristine` command installs every gem fresh from the .gem source, but does not fetch gems from the server or read the source index. `pristine` should be performed after checksums are written using the above commands. +If a user wishes to ensure that the environment they are using contains only gems installed from verified sources that match the checksums in the Gemfile.lock, the command `bundle pristine` should be run. The `bundle pristine` command installs every gem fresh from the .gem source, but does not fetch gems from the server or read the source index. Therefore, if a users wants all gems to be verified upon install, `bundle pristine` should be run after checksums are written to the Gemfile.lock. Example: @@ -79,55 +79,63 @@ If you are sure that the downloaded gem is correct, you can run… Remove the offending line from CHECKSUMS in Gemfile.lock ``` -Certain checksums will always be unavailable because the source does not provide a checksum. When a checksum is not available, a line can be added to the Gemfile.lock CHECKSUMS section that explicitly indicates that no checksum is recorded for that gem version. Future installs of gems with intentionally ignored checksums should not error or warn when the gem is installed. The (RFC TODO) command will allow writing ignored checksum versions. Alternatively, the SHA256 digest of the gem at install time can be recorded, which will allow the user to verify future installs do not deviate from the recorded checksum. +Certain checksums will always be unavailable because the source does not provide a checksum. When a checksum is not available, a line can be added to the Gemfile.lock CHECKSUMS section that explicitly indicates that no checksum is recorded for that gem version. Future installs of gems with intentionally ignored checksums should not error or warn when the gem is installed. The **(RFC TODO)** command will allow idicating that a gem version should ignore checksum verifications. Alternatively, the SHA256 digest of the gem at install time can be recorded allowing the user to verify future installs do not deviate from the recorded checksum. Platforms may pose a problem for Gemfile.lock checksums. If a user has not added one of the platforms where they bundle their app, then checksum verification could be skipped. In order to avoid this security vulnerability, the following should happen automatically. 1. When a gem is installed and no checksum is available in the lockfile, a warning should be printed indicating that unverified gems were installed. 2. When a gem is installed where a checksum is available for the same gem on a different platform, an error should be raised. (i.e. if nokogiri for darwin has a checksum, but nokogiri for musl does not, then an error should be raised). -3. When bundle install is configured with frozen, any gem installed that does not have a checksum (and it was not explicitly ignored) should cause an error. +3. When bundle install is configured with frozen, an error should be raised for any gem installed that does not have a checksum unless the checksum is explicitly ignored. -When platform or missing checksum issues are raised, the error should always indicate what command should be run to fix the error. +When platform or missing checksum issues are raised, the error should always indicate what command or process should be performed to fix the error. -Bundling the app in CI should highlight any platform issues if the CI environment matches the production environment. This will make the bundler upgrade progress more smoothly for users. +Bundling the app on another platform, in CI for example, should highlight any platform issues. This will make the bundler upgrade easier for users. -To encourage uptake of this security feature, it should be enabled by default and any actions that have already fetched checksums should record them in the Gemfile.lock. Progressive upgrading to verified checksum behavior should happen naturally unless explicitly disabled. +To encourage uptake of this security feature, it should be enabled by default. Any actions that fetch authoritative checksums should silently record them in the Gemfile.lock. Progressive upgrading to verified checksum behavior should happen naturally unless explicitly disabled. Gem upgrades in the Gemfile.lock happen in two ways currently. Each way will behave differently. 1. The user explicitly runs `bundle update`. -2. Dependabot, or another automation, changes the version of a locked gem directly in the Gemfile.lock (assuming that dependabot is not upgraded to also change the checksum) +2. Dependabot, or another automation, changes the version of a locked gem directly in the Gemfile.lock (assuming current Dependabot behavior without a corresponding upgrade to match this feature). Explicit gem updates will record the new checksum for the gem and remove any old checksums for gems that are no longer in the Gemfile.lock. -Direct updates will print a warning during install that X number of gem(s) could not be verified. If the bundle is frozen, then the warning will instead print an error and halt gem installation. +Direct updates to the Gemfile.lock will print a warning during install. The warning should say that X number of gem(s) could not be verified. +If the bundle is frozen, then the warning should instead print an error and halt gem installation. # Reference-level explanation -This is the technical portion of the RFC. Explain the design in sufficient detail that: +Gem checksums are fetched from rubygems.org as part of the compact index. The checksums are stored in memory during the bundle process. -- Its interaction with other features is clear. -- It is reasonably clear how the feature would be implemented. -- Corner cases are dissected by example. +If any gems are recorded from a separate source or installed via .gem file, another checksum is recorded and compared with the original. -The section should return to the examples given in the previous section, and explain more fully how the detailed proposal makes those examples work. +If any of the checksums for the same gem name, version and platform differ, an error is raised. -# Drawbacks +When the Gemfile.lock is written, a new CHECKSUMS section is written with all the gems in the bundle and their corresponding checksums. + +Future compatibility for new checksum algorithms is supported by reading and writing existing checksums for installed gems even if the algorithm is unknown. Checksum comparisons take into account the algorithm used and raises errors accordingly. + +Bundler commands that interact with checksums either fetch checksums from sources and update the Gemfile.lock or compare checksums in the Gemfile.lock with the computed digest of the gem being installed. +Internal storage of checksums indexes the checksums by a gem's full name (name-version-platform) and the checksum's algorithm. The source of each checksum is stored with the checksum so that errors can detail where conflicting checksums came from. + +When two sources have a checksum for the same gem full name, the are compared. If they are the same, they are merged into a single record that associates the checksum with all of the sources for that checksum. + +# Drawbacks -Excessive failures +### Excessive failures If checksum verification failures happen more often than expected, it could cause the feature to be ignored or derided as poorly designed and implemented. Bundler should progressively update the Gemfile.lock transparently without too much interaction or excessive warnings and failures. Since checksums verification is just a more strict approach to version verification, this feature fits with the existing expectations of Bundler’s features. This means that checksum verification should be unobtrusive. The feature is not a big deal that needs lots of messaging, and should not fail unless bundler is configured to be strict (in the case of missing checksums) or there is an actual verification failure (in the case of corrupt or malicious gems) -Increased installation time +### Increased installation time Gems will be SHA256 digested during install, adding to the install time. This could be a significant burden for some bundles. Future proposals such as content addressable storage for gem data could address this slowdown, but this is not immediately realistic. -Unverifiable gems +### Unverifiable gems When gems come from private gem servers that do not implement the compact index, checksums will not be available. Bundler must provide configuration that can automatically store digests for gems from sources that don’t supply checksums so that the feature is not a burden to users relying on these sources. An option could be created to configure a specified source to always compute checksums for updated versions. This would mean any change in version from the source would be trusted at the time, but any locked version that is installed later would be verified to match the first install. -GitHub Dependabot version upgrades are at risk of failing. +### GitHub Dependabot version upgrades are at risk of failing. If a user configures CI to use locked gem versions, it would be necessary for Dependabot to write the corresponding checksum line to the Gemfile to prevent every CI build from failing. This could cause many users to disable the feature because it interferes with an existing workflow or causes excessive CI failure notices. This can be addressed by making clear messaging about how to fix this problem. From 829128132d011dedb8c754f76a61e2368a73dcf3 Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Fri, 1 Sep 2023 15:12:16 -0700 Subject: [PATCH 4/9] Updates --- text/0011-gem-checksum-verification.md | 33 ++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/text/0011-gem-checksum-verification.md b/text/0011-gem-checksum-verification.md index ecbc284..1247951 100755 --- a/text/0011-gem-checksum-verification.md +++ b/text/0011-gem-checksum-verification.md @@ -121,6 +121,31 @@ Internal storage of checksums indexes the checksums by a gem's full name (name-v When two sources have a checksum for the same gem full name, the are compared. If they are the same, they are merged into a single record that associates the checksum with all of the sources for that checksum. +### Example CHECKSUM section + +A freshly created Rails 7.0.7.2 app creates the following CHECKSUMS section (snipped for brevity): + +``` +CHECKSUMS + actioncable (7.0.7.2) sha256-a921830a59ee314939955c9fc3b922d2b1f3ebc16fdf062370b9078aa0dc28c5 + actionmailbox (7.0.7.2) sha256-33aeae209fc876c072e5ad28c7ffc16ace533d391368ad6390bb6183c2b27a24 + actionmailer (7.0.7.2) sha256-0e9061159af8c220042b7714a2ba01e2d71d2904f308021ec714793e5f9811a0 + actionpack (7.0.7.2) sha256-c441ff3898bf5827540bcab929d2f5be6e75b64c101513629a3c88e269615561 + actiontext (7.0.7.2) sha256-d29eabbfbf0f084a0bddcfc6bd7e6245e209ec3a1def200e95b670e0cdfba033 + actionview (7.0.7.2) sha256-15ba2612efb484ec80d5b656b4ea16e02d34d3f9980cabc13bd8ac15ccea3f94 + activejob (7.0.7.2) sha256-6d8ebd81d29ce65bb57830640fa2d3f01e4cab0d71714a54c2b13763021023a4 + activemodel (7.0.7.2) sha256-45ba827986065ac273b59cb3b6c9ab3da412beca5d465f1acf7a51fb5bc032b3 + activerecord (7.0.7.2) sha256-425f84edb279c02fe2195eee166b20aabb36f51939087d040fa462859bd6790f + activestorage (7.0.7.2) sha256-8f1d79266f148d74e1cc7fcc91f3f04171e0d10c68f8a31ac95d11644114f4f0 + activesupport (7.0.7.2) sha256-62e01393689c8514a65e2cf8be6f4781d1e6c7d9adc25b1056902d8abd659fee + addressable (2.8.5) sha256-63f0fbcde42edf116d6da98a9437f19dd1692152f1efa3fcc4741e443c772117 + # ... SNIP ... + xpath (3.2.0) sha256-6dfda79d91bb3b949b947ecc5919f042ef2f399b904013eb3ef6d20dd3a4082e + zeitwerk (2.6.11) sha256-ade72f223a75c91f3b02b2c941a57fb697bc443d615f38c28773185e08698dd7 +``` + +During the `rails new` command, `bundle install` pulled all the checksums from the compact index on rubygems.org, then computed checksums for each gem as it was installed. + # Drawbacks ### Excessive failures @@ -139,6 +164,10 @@ When gems come from private gem servers that do not implement the compact index, If a user configures CI to use locked gem versions, it would be necessary for Dependabot to write the corresponding checksum line to the Gemfile to prevent every CI build from failing. This could cause many users to disable the feature because it interferes with an existing workflow or causes excessive CI failure notices. This can be addressed by making clear messaging about how to fix this problem. +### Older versions of Bundler + +Old versions of Bundler should ignore the CHECKSUMS section. We will need to check older versions to be sure. + # Rationale and Alternatives - Rubygems.org already stores SHA256 checksums for gems and returns them in the compact index response. All the information is already present for checksum verification on the client side. @@ -150,3 +179,7 @@ If a user configures CI to use locked gem versions, it would be necessary for De - What are the exact commands, options, warning and error messages that are used to interact with the feature? - What are the unexpected errors that may happen the first time people verify the gems they are downloading? Are there any errors or problems in the existing rubygems.org repository? + +### How do we handle confusion about the authority of checksums written to the Gemfile.lock + +The source of checksums in the Gemfile.lock becomes a matter of trust once it's written. Did the checksum come from the API or was it calculated from a .gem file on a developers computer. If a checksum error is resolved by one developer in a way that saves an incorrect checksum, how should people know when to approve these changes or not. It may not even be common practice for most teams to look at the Gemfile.lock, and changes can often be hidden in pull request reviews. Without a process for checking that the checksums are trustworthy, it's left to every development team to decide on a process. One solution would be a bundle command that could be run in CI every time the gems are installed that verifies the authenticity of checksums in the Gemfile.lock. From e713a07cafce65dea7510f201948ab17f61d0f27 Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Fri, 1 Sep 2023 15:12:39 -0700 Subject: [PATCH 5/9] fix whitespace --- text/0011-gem-checksum-verification.md | 1 - 1 file changed, 1 deletion(-) diff --git a/text/0011-gem-checksum-verification.md b/text/0011-gem-checksum-verification.md index 1247951..45c0726 100755 --- a/text/0011-gem-checksum-verification.md +++ b/text/0011-gem-checksum-verification.md @@ -174,7 +174,6 @@ Old versions of Bundler should ignore the CHECKSUMS section. We will need to che - Verifying .gem files at install time offers nearly complete protection against hacked, altered and corrupted gems. Bundler already ensures that only the bundled gems are available to the app. This feature adds the assurance that only verified gems were installed with the bundle. - One alternative to this solution is including (vendoring) the bundled gems in the repository. This effectively has the same result, since the gems that will be installed during the production deploy will be verifiably the same gems that were used during CI and development. The downside of this vendored approach is the increase in the repository size. Checksums allow for a similar level of confidence without the downsides of vendoring gems and can be enabled by default so that more users will benefit. - # Unresolved questions - What are the exact commands, options, warning and error messages that are used to interact with the feature? From 76794b96039f6a675c72c4e17059201a5eb51bd0 Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Sun, 15 Oct 2023 18:07:54 -0700 Subject: [PATCH 6/9] Update text in response to comments and clarify behaviors --- text/0011-gem-checksum-verification.md | 175 +++++++++++++++---------- 1 file changed, 104 insertions(+), 71 deletions(-) diff --git a/text/0011-gem-checksum-verification.md b/text/0011-gem-checksum-verification.md index 45c0726..61ee3ef 100755 --- a/text/0011-gem-checksum-verification.md +++ b/text/0011-gem-checksum-verification.md @@ -9,19 +9,20 @@ Bundler is adding checksum verification of gems when they are installed. It shou # Motivation -Verifying gem checksums with known checksums at install time is a stronger way to verify that the exact same gem source is being used in every environment where the bundle is installed. Using checksums, especially those sourced from rubygems.org, is a more strict way of locking to a specific gem version. The feature should work transparently in much the same way that bundler already locks to specific versions. +Verifying gem checksums with known checksums at install time is a stronger way to verify that the exact same gem source is being used in every environment where the bundle is installed. +Using checksums, sourced from rubygems.org or from digests of local .gem file, is a more strict way of locking to a specific gem version. +The feature should work transparently in much the same way that bundler already locks to specific versions by ensuring that not just the same version, but the exact same rubygem file data, is installed on each environment. # Guide-level explanation -Upon first upgrading to the version of Bundler with this feature, the checksum feature will be enabled. +Upon first upgrading to this version of Bundler, expect bundler to continue to be secure by default and to progressively add checksums to the lockfile. -First usage will fall into two use cases: -1. Users that want to immediately use the new checksums feature. -2. Users that are not specifically using the checksums feature, but expect bundler to work securely without interference. +If you wish to immediately add checksums to your lockfile for all locked gems, run `bundle lock`. +Bundle lock now fetches checksums from remote sources by default. +If you would like to bypass this behavior, add the `--no-checksums` flag. -For users excited about the new feature, the usual commands and docs should show the new options for securing the bundle with checksums. Documentation and the release notice should include instructions for directly using the new feature. - -For users that are not specifically looking for the feature, common commands like `bundle install`, `bundle update`, `bundle lock`, `bundle add`, should automatically make use of checksums when available. When there are missing checksums that must be fetched with a specific interaction, the UI should say what that action is. +Common Bundler commands like `bundle install`, `bundle update`, `bundle lock`, `bundle add` will now automatically record and verify checksums. +Commands that rely on checksums for verification will output a message or warning when checksums are missing or mismatched. Example: @@ -29,13 +30,16 @@ Example: $ bundle install Bundle complete! 88 Gemfile dependencies, 256 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. -256 gems are missing checksums. Source integrity could not be verified. -Use `bundle lock` to add checksums to Gemfile.lock. +Use `bundle lock` to add missing checksums to Gemfile.lock. ``` -When a gem version is updated in the lockfile using a source that has checksums (like rubygems.org), the checksum should be recorded. Running `bundle update` or `bundle add` should record the checksum from the source into the Gemfile.lock. If a checksum is not available from the source because the source does not provide such info, then a checksum should be created during install using the .gem file or a warning should be printed. +Running `bundle update` or `bundle add` will record the checksum from the source (e.g. rubygems.org) into the Gemfile.lock, if it is available. +If a checksum is not available from the source because the source does not provide such info (e.g. private gemservers) then a checksum will be created during install using the .gem file. +If a checksum can't be created because the source is a path or git source, then only the gems name and version will be recorded with a blank checksum. -If a user wishes to ensure that the environment they are using contains only gems installed from verified sources that match the checksums in the Gemfile.lock, the command `bundle pristine` should be run. The `bundle pristine` command installs every gem fresh from the .gem source, but does not fetch gems from the server or read the source index. Therefore, if a users wants all gems to be verified upon install, `bundle pristine` should be run after checksums are written to the Gemfile.lock. +If you want to ensure that the bundled environment contains only gems matching the checksums in the lockfile, run `bundle pristine`. +The `bundle pristine` command installs every gem fresh from a newly downloaded .gem source. +The pristine install will trigger computation and comparison of the generated SHA256 checksum with the checksum stored in the lockfile. Example: @@ -44,82 +48,72 @@ $ bundle pristine Installing rake 13.0.6 … Installing rspec 3.12.0 -42 gems are missing checksums. Source integrity could not be verified. +42 gems without checksums. Use `bundle lock` to add checksums to Gemfile.lock. -$ bundle lock --update +$ bundle lock Writing lockfile to path/to/Gemfile.lock $ bundle pristine -Installing rake 13.0.6 (verified) +Installing rake 13.0.6 … -Installing rspec 3.12.0 (verified) +Installing rspec 3.12.0 $ ``` -If a gem that is about to be installed does not match the checksum, an error is generated. +When bundler installs a gem from a `.gem` file, it computes the SHA256 checksum of the file. +If an existing checksum is available from the lockfile or the remote source, it will be compared with the computed checksum at install time. +If the checksums do not match, an error is generated and installation is halted. +If no checksum is recorded in the lockfile, the computed checksum is saved to the lockfile where future installs can verify that the same gem is installed. Example: ``` $ bundle install Installing rake 13.0.6 -Bundler cannot continue installing rake-13.0.6. -The checksum for the downloaded `rake-13.0.6.gem` does not match the known checksum for the gem. - -Expected: sha256-814828c34f1315d7e7b7e8295184577cc4e969bad6156ac069d02d63f58d82e8 from: Gemfile.lock:21:1 CHECKSUMS rake (13.0.6) -Downloaded: sha256-69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b69b6 from: -path/to/rake-13.0.6.gem SHA256 hexdigest - -To resolve this issue: -Delete the downloaded gem files referenced above and run `bundle install` - -If you are sure that the downloaded gem is correct, you can run… -(RFC TODO: Currently undecided on what the command should be here to allow unverified gems.) -`bundle install --skip-verification` OR -`bundle config set unverified ignore` OR -Remove the offending line from CHECKSUMS in Gemfile.lock +Bundler found mismatched checksums. This is a potential security risk. + rake (13.0.6) sha256-2222222222222222222222222222222222222222222222222222222222222222 + form the lockfile CHECKSUMS at Gemfile.lock:21:17 + rake (13.0.6) sha256-814828c34f1315d7e7b7e8295184577cc4e969bad6156ac069d02d63f58d82e8 + from the gem at path/to/rake-13.0.6.gem + + To resolve this issue you can either: + 1. remove the gem at path/to/rake-13.0.6.gem + 2. run `bundle install` + or if you are sure that the new checksum from the lockfile CHECKSUMS at Gemfile.lock:21:17 is correct: + 1. remove the matching checksum in Gemfile.lock:21:17 + 2. run `bundle install` + +To ignore checksum security warnings, disable checksum validation with + `bundle config set --local disable_checksum_validation true` ``` -Certain checksums will always be unavailable because the source does not provide a checksum. When a checksum is not available, a line can be added to the Gemfile.lock CHECKSUMS section that explicitly indicates that no checksum is recorded for that gem version. Future installs of gems with intentionally ignored checksums should not error or warn when the gem is installed. The **(RFC TODO)** command will allow idicating that a gem version should ignore checksum verifications. Alternatively, the SHA256 digest of the gem at install time can be recorded allowing the user to verify future installs do not deviate from the recorded checksum. - -Platforms may pose a problem for Gemfile.lock checksums. If a user has not added one of the platforms where they bundle their app, then checksum verification could be skipped. In order to avoid this security vulnerability, the following should happen automatically. - -1. When a gem is installed and no checksum is available in the lockfile, a warning should be printed indicating that unverified gems were installed. -2. When a gem is installed where a checksum is available for the same gem on a different platform, an error should be raised. (i.e. if nokogiri for darwin has a checksum, but nokogiri for musl does not, then an error should be raised). -3. When bundle install is configured with frozen, an error should be raised for any gem installed that does not have a checksum unless the checksum is explicitly ignored. - -When platform or missing checksum issues are raised, the error should always indicate what command or process should be performed to fix the error. - -Bundling the app on another platform, in CI for example, should highlight any platform issues. This will make the bundler upgrade easier for users. - -To encourage uptake of this security feature, it should be enabled by default. Any actions that fetch authoritative checksums should silently record them in the Gemfile.lock. Progressive upgrading to verified checksum behavior should happen naturally unless explicitly disabled. +Certain checksums will always be unavailable because the source does not provide a checksum. +When a checksum is not available, the gem will be added to the Gemfile.lock CHECKSUMS section without a checksum. -Gem upgrades in the Gemfile.lock happen in two ways currently. Each way will behave differently. - -1. The user explicitly runs `bundle update`. -2. Dependabot, or another automation, changes the version of a locked gem directly in the Gemfile.lock (assuming current Dependabot behavior without a corresponding upgrade to match this feature). - -Explicit gem updates will record the new checksum for the gem and remove any old checksums for gems that are no longer in the Gemfile.lock. - -Direct updates to the Gemfile.lock will print a warning during install. The warning should say that X number of gem(s) could not be verified. -If the bundle is frozen, then the warning should instead print an error and halt gem installation. +Users should be aware that when bundling on CI or production, new gems can be added for a platform not found in the Gemfile.lock. +Bundler will silently record the new checksums for the missing gem just like on a local development machine. +If you would like to ensure that only lockfile checksums are used, bundle install should use use the frozen or deployment configurations. # Reference-level explanation -Gem checksums are fetched from rubygems.org as part of the compact index. The checksums are stored in memory during the bundle process. +Gem checksums are fetched from rubygems.org as part of the compact index. +The checksums are stored in memory during the bundle process. If any gems are recorded from a separate source or installed via .gem file, another checksum is recorded and compared with the original. -If any of the checksums for the same gem name, version and platform differ, an error is raised. +If any of the checksums for the same gem name, version and platform differ, an error is raised with instructions for resolving the error. When the Gemfile.lock is written, a new CHECKSUMS section is written with all the gems in the bundle and their corresponding checksums. -Future compatibility for new checksum algorithms is supported by reading and writing existing checksums for installed gems even if the algorithm is unknown. Checksum comparisons take into account the algorithm used and raises errors accordingly. +Future compatibility for new checksum algorithms is supported by reading and writing existing checksums for installed gems even if the algorithm is unknown. +Checksum comparisons take into account the algorithm used and raises errors accordingly. Bundler commands that interact with checksums either fetch checksums from sources and update the Gemfile.lock or compare checksums in the Gemfile.lock with the computed digest of the gem being installed. -Internal storage of checksums indexes the checksums by a gem's full name (name-version-platform) and the checksum's algorithm. The source of each checksum is stored with the checksum so that errors can detail where conflicting checksums came from. +Internal storage of checksums indexes the checksums by a gem's NameTuple (name, version, platform) and the checksum's algorithm. +The source of each checksum is stored with the checksum so that errors can describe how to fix conflicting checksums. -When two sources have a checksum for the same gem full name, the are compared. If they are the same, they are merged into a single record that associates the checksum with all of the sources for that checksum. +When two remote sources have a checksum for the same gem, the are compared. +If they are the same, bundler proceeds as normal ### Example CHECKSUM section @@ -150,35 +144,74 @@ During the `rails new` command, `bundle install` pulled all the checksums from t ### Excessive failures -If checksum verification failures happen more often than expected, it could cause the feature to be ignored or derided as poorly designed and implemented. Bundler should progressively update the Gemfile.lock transparently without too much interaction or excessive warnings and failures. Since checksums verification is just a more strict approach to version verification, this feature fits with the existing expectations of Bundler’s features. This means that checksum verification should be unobtrusive. The feature is not a big deal that needs lots of messaging, and should not fail unless bundler is configured to be strict (in the case of missing checksums) or there is an actual verification failure (in the case of corrupt or malicious gems) +If checksum verification failures happen more often than expected, it could cause the feature to be ignored or derided as poorly designed and implemented. +Bundler should progressively update the Gemfile.lock transparently without too much interaction or excessive warnings and failures. +Checksums verification is just a more strict approach to version verification, so this feature fits with the existing expectations of Bundler’s features and should be unobtrusive. +The feature is not a "big deal" that needs lots of warnings or errors to encourage usage. +It should not fail unless bundler is configured to be strict (a frozen bundle) or there is an actual verification failure (a corrupt or malicious gem). ### Increased installation time -Gems will be SHA256 digested during install, adding to the install time. This could be a significant burden for some bundles. Future proposals such as content addressable storage for gem data could address this slowdown, but this is not immediately realistic. +Gems will be SHA256 digested during install, slightly adding to the install time. +This could be a larger burden for some bundles on slower machines. + +Future proposals could attempt to address this slowdown, if necessary. ### Unverifiable gems -When gems come from private gem servers that do not implement the compact index, checksums will not be available. Bundler must provide configuration that can automatically store digests for gems from sources that don’t supply checksums so that the feature is not a burden to users relying on these sources. An option could be created to configure a specified source to always compute checksums for updated versions. This would mean any change in version from the source would be trusted at the time, but any locked version that is installed later would be verified to match the first install. +When gems come from private gem servers that do not implement the compact index, checksums will not be available. +Bundler will calculate digests from .gem files from sources that don’t supply checksums. + +An additional flag for `bundle lock` could be provided that allows reinstalling, and thus calculating the checksum for, gems that don't have a checksum. + +Gems from a git source are verified by nature of matching a specific git SHA and should be excluded from checksum verification. -### GitHub Dependabot version upgrades are at risk of failing. +Path source gems cannot be verified because the checksum of the entire path would be complicated to calculate and unreliable. -If a user configures CI to use locked gem versions, it would be necessary for Dependabot to write the corresponding checksum line to the Gemfile to prevent every CI build from failing. This could cause many users to disable the feature because it interferes with an existing workflow or causes excessive CI failure notices. This can be addressed by making clear messaging about how to fix this problem. +### GitHub Dependabot and other automations are at risk of failing. + +If a user configures CI to install with frozen bundle, then all dependabot pull requests will fail. +This will make dependabot update branches much less useful, since a failing update is almost guaranteed. +In order to remain functional, Dependabot would need to write the corresponding checksum lines to the Gemfile to prevent every CI build from failing. +This would make dependabot the defacto source of the checksums in the Gemfile instead of rubygems.org or .gem files. +This might lead users to disable frozen bundles, disable dependabot, or disable the checksums feature, any of which could be considered major drawbacks. + +This can be addressed by making clear messaging about how to fix this problem (checkout the branch and run bundle install) +Additionally, we should work with the dependabot team to provide the information necessary to add checksums to the lockfile. ### Older versions of Bundler -Old versions of Bundler should ignore the CHECKSUMS section. We will need to check older versions to be sure. +Old versions of Bundler should ignore the CHECKSUMS section. # Rationale and Alternatives -- Rubygems.org already stores SHA256 checksums for gems and returns them in the compact index response. All the information is already present for checksum verification on the client side. -- Verifying .gem files at install time offers nearly complete protection against hacked, altered and corrupted gems. Bundler already ensures that only the bundled gems are available to the app. This feature adds the assurance that only verified gems were installed with the bundle. -- One alternative to this solution is including (vendoring) the bundled gems in the repository. This effectively has the same result, since the gems that will be installed during the production deploy will be verifiably the same gems that were used during CI and development. The downside of this vendored approach is the increase in the repository size. Checksums allow for a similar level of confidence without the downsides of vendoring gems and can be enabled by default so that more users will benefit. +Rubygems.org already stores SHA256 checksums for gems and returns them in the compact index response. +All the information is already present for checksum verification on the client side. + +Verifying .gem files at install time offers nearly complete protection against hacked, altered and corrupted gems. +Bundler already ensures that only the bundled gems are available to the app. +This feature adds the assurance that only verified gems were installed with the bundle. + +One alternative to this solution is including (vendoring) the bundled gems in the repository. +This effectively has the same result, since the gems that will be installed during the production deploy will be verifiably the same gems that were used during CI and development. +The downside of this vendored approach is the increase in the repository size. +The upside is installation that doesn't depend on a remote source. +Checksums allow for a similar level of confidence without the larger repository size of vendoring gems. +It can be enabled by default so that more users will benefit. # Unresolved questions -- What are the exact commands, options, warning and error messages that are used to interact with the feature? -- What are the unexpected errors that may happen the first time people verify the gems they are downloading? Are there any errors or problems in the existing rubygems.org repository? +### What are the unexpected errors that may happen the first time developers interact with this feature? + +In particular, the first deploy for users with a frozen bundle in CI and/or production may produce errors that might not have obvious solutions to someone unfamiliar with the feature. ### How do we handle confusion about the authority of checksums written to the Gemfile.lock -The source of checksums in the Gemfile.lock becomes a matter of trust once it's written. Did the checksum come from the API or was it calculated from a .gem file on a developers computer. If a checksum error is resolved by one developer in a way that saves an incorrect checksum, how should people know when to approve these changes or not. It may not even be common practice for most teams to look at the Gemfile.lock, and changes can often be hidden in pull request reviews. Without a process for checking that the checksums are trustworthy, it's left to every development team to decide on a process. One solution would be a bundle command that could be run in CI every time the gems are installed that verifies the authenticity of checksums in the Gemfile.lock. +The source of checksums in the Gemfile.lock becomes a matter of trust once it's written. +Did the checksum come from the API or was it calculated from a .gem file on a developers computer. +If a checksum error is resolved by one developer in a way that saves an incorrect checksum, how should people know when to approve these changes or not. +It may not be common practice for most teams to look at the Gemfile.lock when reviewing code. +Gemfile.lock changes can be hidden in pull request reviews. +Without a process for checking that the checksums are trustworthy, it's left to every development team to decide on a process. + +One solution would be a bundle command that could be run in CI every time the gems are installed that verifies the authenticity of checksums in the Gemfile.lock. From 7c9b3e83d2e7b4b9dfc5d748ee19deda08afd28c Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Fri, 20 Oct 2023 22:04:26 -0700 Subject: [PATCH 7/9] Update 0011-gem-checksum-verification.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: David Rodríguez --- text/0011-gem-checksum-verification.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0011-gem-checksum-verification.md b/text/0011-gem-checksum-verification.md index 61ee3ef..88a50cd 100755 --- a/text/0011-gem-checksum-verification.md +++ b/text/0011-gem-checksum-verification.md @@ -112,7 +112,7 @@ Bundler commands that interact with checksums either fetch checksums from source Internal storage of checksums indexes the checksums by a gem's NameTuple (name, version, platform) and the checksum's algorithm. The source of each checksum is stored with the checksum so that errors can describe how to fix conflicting checksums. -When two remote sources have a checksum for the same gem, the are compared. +When two remote sources have a checksum for the same gem, they are compared. If they are the same, bundler proceeds as normal ### Example CHECKSUM section From 59224ad2f00a3372154b76b138015429b3c64455 Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Sun, 22 Oct 2023 20:05:32 -0700 Subject: [PATCH 8/9] Updates to clarify after feedback --- text/0011-gem-checksum-verification.md | 68 ++++++++++++++------------ 1 file changed, 37 insertions(+), 31 deletions(-) diff --git a/text/0011-gem-checksum-verification.md b/text/0011-gem-checksum-verification.md index 88a50cd..64ff572 100755 --- a/text/0011-gem-checksum-verification.md +++ b/text/0011-gem-checksum-verification.md @@ -15,14 +15,14 @@ The feature should work transparently in much the same way that bundler already # Guide-level explanation -Upon first upgrading to this version of Bundler, expect bundler to continue to be secure by default and to progressively add checksums to the lockfile. - -If you wish to immediately add checksums to your lockfile for all locked gems, run `bundle lock`. -Bundle lock now fetches checksums from remote sources by default. -If you would like to bypass this behavior, add the `--no-checksums` flag. +Upon first upgrading to this version of Bundler, expect bundler to progressively add checksums to the lockfile without any extra configuration. Common Bundler commands like `bundle install`, `bundle update`, `bundle lock`, `bundle add` will now automatically record and verify checksums. -Commands that rely on checksums for verification will output a message or warning when checksums are missing or mismatched. +Commands that rely on checksums for verification will silently succeed when checksums match and fail with a unique non-zero exit code when checksums do not match. + +If you wish to immediately add all available checksums to your lockfile for your bundled gems, run `bundle lock`. +Bundle lock now fetches checksums from remote sources by default. +If you would like to bypass this behavior, run `bundle lock --no-checksums`. Example: @@ -30,7 +30,6 @@ Example: $ bundle install Bundle complete! 88 Gemfile dependencies, 256 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. -Use `bundle lock` to add missing checksums to Gemfile.lock. ``` Running `bundle update` or `bundle add` will record the checksum from the source (e.g. rubygems.org) into the Gemfile.lock, if it is available. @@ -70,15 +69,15 @@ Example: $ bundle install Installing rake 13.0.6 Bundler found mismatched checksums. This is a potential security risk. - rake (13.0.6) sha256-2222222222222222222222222222222222222222222222222222222222222222 + rake (13.0.6) sha256=2222222222222222222222222222222222222222222222222222222222222222 form the lockfile CHECKSUMS at Gemfile.lock:21:17 - rake (13.0.6) sha256-814828c34f1315d7e7b7e8295184577cc4e969bad6156ac069d02d63f58d82e8 + rake (13.0.6) sha256=814828c34f1315d7e7b7e8295184577cc4e969bad6156ac069d02d63f58d82e8 from the gem at path/to/rake-13.0.6.gem To resolve this issue you can either: 1. remove the gem at path/to/rake-13.0.6.gem 2. run `bundle install` - or if you are sure that the new checksum from the lockfile CHECKSUMS at Gemfile.lock:21:17 is correct: + or if you are sure that the new checksum from the gem at path/to/rake-13.0.6 is correct: 1. remove the matching checksum in Gemfile.lock:21:17 2. run `bundle install` @@ -121,21 +120,21 @@ A freshly created Rails 7.0.7.2 app creates the following CHECKSUMS section (sni ``` CHECKSUMS - actioncable (7.0.7.2) sha256-a921830a59ee314939955c9fc3b922d2b1f3ebc16fdf062370b9078aa0dc28c5 - actionmailbox (7.0.7.2) sha256-33aeae209fc876c072e5ad28c7ffc16ace533d391368ad6390bb6183c2b27a24 - actionmailer (7.0.7.2) sha256-0e9061159af8c220042b7714a2ba01e2d71d2904f308021ec714793e5f9811a0 - actionpack (7.0.7.2) sha256-c441ff3898bf5827540bcab929d2f5be6e75b64c101513629a3c88e269615561 - actiontext (7.0.7.2) sha256-d29eabbfbf0f084a0bddcfc6bd7e6245e209ec3a1def200e95b670e0cdfba033 - actionview (7.0.7.2) sha256-15ba2612efb484ec80d5b656b4ea16e02d34d3f9980cabc13bd8ac15ccea3f94 - activejob (7.0.7.2) sha256-6d8ebd81d29ce65bb57830640fa2d3f01e4cab0d71714a54c2b13763021023a4 - activemodel (7.0.7.2) sha256-45ba827986065ac273b59cb3b6c9ab3da412beca5d465f1acf7a51fb5bc032b3 - activerecord (7.0.7.2) sha256-425f84edb279c02fe2195eee166b20aabb36f51939087d040fa462859bd6790f - activestorage (7.0.7.2) sha256-8f1d79266f148d74e1cc7fcc91f3f04171e0d10c68f8a31ac95d11644114f4f0 - activesupport (7.0.7.2) sha256-62e01393689c8514a65e2cf8be6f4781d1e6c7d9adc25b1056902d8abd659fee - addressable (2.8.5) sha256-63f0fbcde42edf116d6da98a9437f19dd1692152f1efa3fcc4741e443c772117 + actioncable (7.0.7.2) sha256=a921830a59ee314939955c9fc3b922d2b1f3ebc16fdf062370b9078aa0dc28c5 + actionmailbox (7.0.7.2) sha256=33aeae209fc876c072e5ad28c7ffc16ace533d391368ad6390bb6183c2b27a24 + actionmailer (7.0.7.2) sha256=0e9061159af8c220042b7714a2ba01e2d71d2904f308021ec714793e5f9811a0 + actionpack (7.0.7.2) sha256=c441ff3898bf5827540bcab929d2f5be6e75b64c101513629a3c88e269615561 + actiontext (7.0.7.2) sha256=d29eabbfbf0f084a0bddcfc6bd7e6245e209ec3a1def200e95b670e0cdfba033 + actionview (7.0.7.2) sha256=15ba2612efb484ec80d5b656b4ea16e02d34d3f9980cabc13bd8ac15ccea3f94 + activejob (7.0.7.2) sha256=6d8ebd81d29ce65bb57830640fa2d3f01e4cab0d71714a54c2b13763021023a4 + activemodel (7.0.7.2) sha256=45ba827986065ac273b59cb3b6c9ab3da412beca5d465f1acf7a51fb5bc032b3 + activerecord (7.0.7.2) sha256=425f84edb279c02fe2195eee166b20aabb36f51939087d040fa462859bd6790f + activestorage (7.0.7.2) sha256=8f1d79266f148d74e1cc7fcc91f3f04171e0d10c68f8a31ac95d11644114f4f0 + activesupport (7.0.7.2) sha256=62e01393689c8514a65e2cf8be6f4781d1e6c7d9adc25b1056902d8abd659fee + addressable (2.8.5) sha256=63f0fbcde42edf116d6da98a9437f19dd1692152f1efa3fcc4741e443c772117 # ... SNIP ... - xpath (3.2.0) sha256-6dfda79d91bb3b949b947ecc5919f042ef2f399b904013eb3ef6d20dd3a4082e - zeitwerk (2.6.11) sha256-ade72f223a75c91f3b02b2c941a57fb697bc443d615f38c28773185e08698dd7 + xpath (3.2.0) sha256=6dfda79d91bb3b949b947ecc5919f042ef2f399b904013eb3ef6d20dd3a4082e + zeitwerk (2.6.11) sha256=ade72f223a75c91f3b02b2c941a57fb697bc443d615f38c28773185e08698dd7 ``` During the `rails new` command, `bundle install` pulled all the checksums from the compact index on rubygems.org, then computed checksums for each gem as it was installed. @@ -170,14 +169,21 @@ Path source gems cannot be verified because the checksum of the entire path woul ### GitHub Dependabot and other automations are at risk of failing. -If a user configures CI to install with frozen bundle, then all dependabot pull requests will fail. -This will make dependabot update branches much less useful, since a failing update is almost guaranteed. -In order to remain functional, Dependabot would need to write the corresponding checksum lines to the Gemfile to prevent every CI build from failing. -This would make dependabot the defacto source of the checksums in the Gemfile instead of rubygems.org or .gem files. -This might lead users to disable frozen bundles, disable dependabot, or disable the checksums feature, any of which could be considered major drawbacks. +In the normal case with a non-frozen bundle, Dependabot will continue to work as normal. +Dependabot will not immediately add checksums to Dependabot PRs, but this is not an error. +The next user install after a dependabot commit will add the checksum. + +However, if CI uses a frozen bundle, then all dependabot pull requests will fail due to missing checksums. +This should be addressed by printing clear messaging about how to fix this problem: checkout the branch and run `bundle install`. + +Until patched, dependabot update branches will be less useful because they will require manual intervention before the build will work. +This might lead users to disable frozen bundles or disable the checksums feature to avoid the CI failures and continue with their existing workflow. +In this case, leaving these features disabled, maybe for longer than necessary, could impact the security of these user's application. + +Dependabot maintainers will need to update Dependabot to write the corresponding checksum to the Gemfile to prevent CI build failures caused by frozen bundle missing checksums. +This makes Dependabot the defacto source of the checksums in the Gemfile for updated gems, which is hopefully already clear to users merging these PRs and should be considered part of the trust extended to GitHub already. -This can be addressed by making clear messaging about how to fix this problem (checkout the branch and run bundle install) -Additionally, we should work with the dependabot team to provide the information necessary to add checksums to the lockfile. +We are open to working with the dependabot team to provide the information necessary to add checksums to the lockfile. ### Older versions of Bundler From 27c23fc75a9af48657b15d325e9d17ac13d1e739 Mon Sep 17 00:00:00 2001 From: Martin Emde Date: Wed, 22 Nov 2023 15:54:19 -0800 Subject: [PATCH 9/9] Change intro to mention opt-in --- text/0011-gem-checksum-verification.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0011-gem-checksum-verification.md b/text/0011-gem-checksum-verification.md index 64ff572..9920bfe 100755 --- a/text/0011-gem-checksum-verification.md +++ b/text/0011-gem-checksum-verification.md @@ -15,7 +15,9 @@ The feature should work transparently in much the same way that bundler already # Guide-level explanation -Upon first upgrading to this version of Bundler, expect bundler to progressively add checksums to the lockfile without any extra configuration. +Upon first upgrading to this version of Bundler, lockfile checksums will not enabled. +The feature will opt-in only at initial release. +It is important that Bundler maintains compatibility and there are edge cases that are difficult to accommodate without wider adoption. Common Bundler commands like `bundle install`, `bundle update`, `bundle lock`, `bundle add` will now automatically record and verify checksums. Commands that rely on checksums for verification will silently succeed when checksums match and fail with a unique non-zero exit code when checksums do not match.