From 76e89259f94cf676c1c28e3383fb7e3e3c2f0111 Mon Sep 17 00:00:00 2001 From: Asaf Algawi Date: Thu, 12 Sep 2024 10:34:01 +0000 Subject: [PATCH 1/3] Create proposal for annotation based filtering --- .../Annotation-Based-Artifact-Filtration.md | 131 ++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 docs/proposals/Annotation-Based-Artifact-Filtration.md diff --git a/docs/proposals/Annotation-Based-Artifact-Filtration.md b/docs/proposals/Annotation-Based-Artifact-Filtration.md new file mode 100644 index 000000000..9914d3cde --- /dev/null +++ b/docs/proposals/Annotation-Based-Artifact-Filtration.md @@ -0,0 +1,131 @@ +# Annotation Based Artifact Filteration + +## Problem/Motivation + +When configuring a verifier in Ratify, we set the artifact type the verifier should work on. In such case, Ratify will verify all referrers of a given subject that have a matching artifact type using the verifier. +In some cases, this could lead to a wrong behavior. For instance, Vulnerability Artifacts are outdated once a new artifact is written to the repository, as such there is no use for verifying both the new one and the old one. On the other hand, in performing signature verification using Notary, we may wish to attest more than one signature, because there may be multiple signatures attached to a given artifact, but our "tenant" only trusts a subset of them. + +The issue with verifying all the matching artifacts could also lead to performance issues, each "verification" process hides within a request to pull the artifcat manifest, and the blobs containing the actual data. +In previous studies made by the ratify team, it was observed that opverloading the registry with requests could lead to errors and throtteling. (see: https://ratify.dev/docs/reference/performance) + +Given the performance study listed above, in order to provide the best experience for Ratify's users ratify would reduce the load it generates on an the registry, thus reducing the chance for throtteling. + +# Proposed Solution + +Ratify uses the Referrer API (see: https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers) in order to obtain the list of attached artifacts of a given subject artifact. The response body for this request is a generated OCI image index, that looks like this: + +```json +{ + "schemaVersion": 2, + "mediaType": "application/vnd.oci.image.index.v1+json", + "manifests": [ + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 1234, + "digest": "sha256:a1a1a1...", + "artifactType": "application/vnd.example.sbom.v1", + "annotations": { + "org.opencontainers.image.created": "2022-01-01T14:42:55Z", + "org.example.sbom.format": "json" + } + }, + { + "mediaType": "application/vnd.oci.image.manifest.v1+json", + "size": 1234, + "digest": "sha256:a2a2a2...", + "artifactType": "application/vnd.example.signature.v1", + "annotations": { + "org.opencontainers.image.created": "2022-01-01T07:21:33Z", + "org.example.signature.fingerprint": "abcd" + } + }, + { + "mediaType": "application/vnd.oci.image.index.v1+json", + "size": 1234, + "digest": "sha256:a3a3a3...", + "annotations": { + "org.opencontainers.image.created": "2023-01-01T07:21:33Z", + } + } + ] +} +``` + +The image index response is requried by API to include the image annotations, which gives Ratify to oprrotunity to perform some basic filtration before invoking the verifier on the listed artifacts. The exact mechanism for requiring the filtration should be specific to each verifier as the behavior and logic differs based on is actually being attested, as such, it will be part of the verifier configuration. + +# Filtration Methods + +## Age Based Filtration + +Assuming artifacts are generated by ORAS, they all have a `org.opencontainers.image.created` annotation, that markes the creation date of the artifact. Based on it, Ratify could filter out stale artifacts and only evaluate the latest image. To achieve this, Ratify would have to read all the referrers, ordering them based on the artifact age, and only pass the latest one to the corresponding verifier. + +This kind of filtering strategy is best used on artifacts that are rapidly changing, for example Vulnerability Assessment artifacts that are immedietly oudated once a new artifact is pushed to the registry. + +## Annotation Value based filteration + +Ratify could perform string comparison checks in order to evaluate if a given arifact should be evaluted by a given verifier or not, the flitering would be made by checking if a given annotation +is part of a list of approved annotation values. + +To illustrate an example usage of this kind of filtration we could take a look at the following notary artifact manifest: + +```json +{ + "schemaVersion": 2, + "manifests": [ + { + "mediaType": "application/vnd.oci.image.config.v1+json", + "size": 7023, + "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b864", + "signatures": [ + { + "mediaType": "applicaiton/vnd.cncf.notary.x509-signature.config.v2", + "digest": "sha256:134c7fe821b9d359490cd009ce7ca322453f4f2d0", + "size": 1337, + "annotations": { + "org.opencontainers.image.signature.fingerprint": "43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8" + } + } + ] + } + ] +} +``` + +We could define a notation verifier configuration that would require the artifact manifest to contain the annotation `org.opencontainers.image.signature.fingerprint` and that is should have a known fingerprint value. this would enable a scenario where the user configures three notation verifiers, one in each namespace: +* Namepsace A -> Fingerprint should match [X] +* Namespace B -> Fingerprint should match [Y] +* Namespace C -> Fingerprint should match either [X, Y] + +## User experiences + +This section describes the experience that users interact with Ratify using the proposed solution. In summary, the propsed solution suggest we should allow for filtering of artifact based on annotations, and as such the following section describes how the customer would configure the filtration. + +Seeing that filteration is unique to each verifier, it should be configured in the verifier itself, as such, in order to maintain backwards compatability, it is important to note that if no filteration is configured the default behavior would be to evaluate all artifacts. + +### Defining artifact age based filtering + +To support artifact age based filtering, we would add an additional field to the verifier configuration: + +```yaml +apiVersion: config.ratify.deislabs.io/v1beta1 +kind: Verifier # NamespacedVerifier has the same spec. +metadata: + name: test-verifier +spec: + name: # REQUIRED: [string], the unique type of the verifier (notation, cosign) + artifactType: # REQUIRED: [string], comma seperated list, artifact type this verifier handles + artifactAgeFilter: # Optional: [string], one of the following 'latest' to verify only the last artifact, or all to verify all artifacts. defaults to all. + artifactAnnotationFilterStrategy: # Optional: [Object], object defining annotation list filtering, when not defined, no filtering is done. + - annotation: # Required: [String], the annotation key + - values: # Required: [[]string], the list of allowed values. + address: # OPTIONAL: [string], Plugin path, defaults to value of env "RATIFY_CONFIG" or "~/.ratify/plugins" + version: # OPTIONAL: [string], Version of the external plugin, defaults to 1.0.0. On ratify initialization, the specified version will be validated against the supported plugin version. + source: + artifact: # OPTIONAL: [string], Source location to download the plugin binary, learn more at docs/reference/dynamic-plugins.md e.g. wabbitnetworks.azurecr.io/test sample-verifier-plugin:v1 + parameters: # OPTIONAL: [object] Parameters specific to this verifier +``` + +# References + +* [Ratify Performance at Scale Study](https://ratify.dev/docs/reference/performance) +* [Referrer API in Distribution Spec](https://github.com/opencontainers/distribution-spec/blob/main/spec.md#listing-referrers) \ No newline at end of file From bcbabffb0e50b0773f391fb6f9438b3ed47df096 Mon Sep 17 00:00:00 2001 From: Asaf Algawi Date: Tue, 8 Oct 2024 15:51:15 +0000 Subject: [PATCH 2/3] updated proposal --- ...on.md => Filter-Artifacts-Based-On-Age.md} | 51 ++++--------------- 1 file changed, 11 insertions(+), 40 deletions(-) rename docs/proposals/{Annotation-Based-Artifact-Filtration.md => Filter-Artifacts-Based-On-Age.md} (66%) diff --git a/docs/proposals/Annotation-Based-Artifact-Filtration.md b/docs/proposals/Filter-Artifacts-Based-On-Age.md similarity index 66% rename from docs/proposals/Annotation-Based-Artifact-Filtration.md rename to docs/proposals/Filter-Artifacts-Based-On-Age.md index 9914d3cde..ad3ec31ac 100644 --- a/docs/proposals/Annotation-Based-Artifact-Filtration.md +++ b/docs/proposals/Filter-Artifacts-Based-On-Age.md @@ -1,9 +1,9 @@ -# Annotation Based Artifact Filteration +# Fitler Artifacts Based On Age ## Problem/Motivation When configuring a verifier in Ratify, we set the artifact type the verifier should work on. In such case, Ratify will verify all referrers of a given subject that have a matching artifact type using the verifier. -In some cases, this could lead to a wrong behavior. For instance, Vulnerability Artifacts are outdated once a new artifact is written to the repository, as such there is no use for verifying both the new one and the old one. On the other hand, in performing signature verification using Notary, we may wish to attest more than one signature, because there may be multiple signatures attached to a given artifact, but our "tenant" only trusts a subset of them. +In some cases, this could lead to a wrong behavior. For instance, Vulnerability Artifacts are outdated once a new artifact is written to the repository, as such there is no use for verifying both the new one and the old one. The issue with verifying all the matching artifacts could also lead to performance issues, each "verification" process hides within a request to pull the artifcat manifest, and the blobs containing the actual data. In previous studies made by the ratify team, it was observed that opverloading the registry with requests could lead to errors and throtteling. (see: https://ratify.dev/docs/reference/performance) @@ -61,41 +61,6 @@ Assuming artifacts are generated by ORAS, they all have a `org.opencontainers.im This kind of filtering strategy is best used on artifacts that are rapidly changing, for example Vulnerability Assessment artifacts that are immedietly oudated once a new artifact is pushed to the registry. -## Annotation Value based filteration - -Ratify could perform string comparison checks in order to evaluate if a given arifact should be evaluted by a given verifier or not, the flitering would be made by checking if a given annotation -is part of a list of approved annotation values. - -To illustrate an example usage of this kind of filtration we could take a look at the following notary artifact manifest: - -```json -{ - "schemaVersion": 2, - "manifests": [ - { - "mediaType": "application/vnd.oci.image.config.v1+json", - "size": 7023, - "digest": "sha256:b5b2b2c507a0944348e0303114d8d93aaaa081732b864", - "signatures": [ - { - "mediaType": "applicaiton/vnd.cncf.notary.x509-signature.config.v2", - "digest": "sha256:134c7fe821b9d359490cd009ce7ca322453f4f2d0", - "size": 1337, - "annotations": { - "org.opencontainers.image.signature.fingerprint": "43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8" - } - } - ] - } - ] -} -``` - -We could define a notation verifier configuration that would require the artifact manifest to contain the annotation `org.opencontainers.image.signature.fingerprint` and that is should have a known fingerprint value. this would enable a scenario where the user configures three notation verifiers, one in each namespace: -* Namepsace A -> Fingerprint should match [X] -* Namespace B -> Fingerprint should match [Y] -* Namespace C -> Fingerprint should match either [X, Y] - ## User experiences This section describes the experience that users interact with Ratify using the proposed solution. In summary, the propsed solution suggest we should allow for filtering of artifact based on annotations, and as such the following section describes how the customer would configure the filtration. @@ -115,9 +80,6 @@ spec: name: # REQUIRED: [string], the unique type of the verifier (notation, cosign) artifactType: # REQUIRED: [string], comma seperated list, artifact type this verifier handles artifactAgeFilter: # Optional: [string], one of the following 'latest' to verify only the last artifact, or all to verify all artifacts. defaults to all. - artifactAnnotationFilterStrategy: # Optional: [Object], object defining annotation list filtering, when not defined, no filtering is done. - - annotation: # Required: [String], the annotation key - - values: # Required: [[]string], the list of allowed values. address: # OPTIONAL: [string], Plugin path, defaults to value of env "RATIFY_CONFIG" or "~/.ratify/plugins" version: # OPTIONAL: [string], Version of the external plugin, defaults to 1.0.0. On ratify initialization, the specified version will be validated against the supported plugin version. source: @@ -125,6 +87,15 @@ spec: parameters: # OPTIONAL: [object] Parameters specific to this verifier ``` +### Considerations +To implmenet age based filteration, Ratify has to be aware of all the attached artifacts of a givan kind before handing them to the verifier that wishes to attest only the latest artifact. In order to implement such behavior some modification has to be made to the executor of Ratify and the verifier implementation. + +Below are two proposals which are currently being considered for implemetation. +| Approach | Pros | Cons| Notes | +| -------- | ---- | ----| ----- | +| 1. Obtain and store all the referrer list
2. Sort it in descending order.
3. Use the CanVerify method of the referrer to make sure a verifier
that only wants the latest artifact is invoked once. | Naive implementation.

Does not make a huge change in the executor, other than fetching the list before hand.

Transparent change for verifiers that do not wish to use verify the latest image only. | The referrer list can be of any arbitrary size, therefore fetching the entire list may cause Ratify to hit a hard memory limit and crash.
To implement the feature with this kind of behavior, Ratify would have to limit the number of attached artifacts it supports to some constant number which will be determined during the implementation.

Additional latency for sorting the artifacts. | A test index list, with ~1000 artifacts within and two annotations (created timestamp, and another text field) weighs around ~400K, default ratify installation has 512MB of ram, so we're well within the limits of 'normal' use. +| 1. Split verifiers into two groups, those which require only latest artifact, and those which operate on all artifacts.
2. For verifiers that work on all artifacts, no change will be made.
3. For verifiers that require only the last artifact, the executor will manage a map between the verifier and an artifact descriptor that is the "current candidate" for being the latest.
4. As we iterate all the referrer, the cadndiate is constantly being update, if a new artifact is discovered.
Once the executor had finished iterating over the referrer list, it would execute all the referrer that required the latest artifact against the "current candidate" which is promised to be latest artifact. | Does not modify Ratify's current scalability.| Requires the executor to be aware of verifier type, possibly by an interface change on the verifier API

Requires changes in multiple places in the executor, performing the verifier loop another time for the second list of verifiers that only require latest artifact. | The benefit of not pulling all the referrers, from the standpoint of keeping the same 'memory footprint' is not clear. + # References * [Ratify Performance at Scale Study](https://ratify.dev/docs/reference/performance) From c61c6466605d6db51cb6d63c95c9551ec4db238d Mon Sep 17 00:00:00 2001 From: Asaf Algawi Date: Tue, 8 Oct 2024 16:09:31 +0000 Subject: [PATCH 3/3] renamed proposal --- ...d-On-Age.md => Verify-Latest-N-Artifacts.md} | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) rename docs/proposals/{Filter-Artifacts-Based-On-Age.md => Verify-Latest-N-Artifacts.md} (78%) diff --git a/docs/proposals/Filter-Artifacts-Based-On-Age.md b/docs/proposals/Verify-Latest-N-Artifacts.md similarity index 78% rename from docs/proposals/Filter-Artifacts-Based-On-Age.md rename to docs/proposals/Verify-Latest-N-Artifacts.md index ad3ec31ac..efe7240eb 100644 --- a/docs/proposals/Filter-Artifacts-Based-On-Age.md +++ b/docs/proposals/Verify-Latest-N-Artifacts.md @@ -1,4 +1,4 @@ -# Fitler Artifacts Based On Age +# Verify only the latest N artifacts ## Problem/Motivation @@ -53,14 +53,15 @@ Ratify uses the Referrer API (see: https://github.com/opencontainers/distributio The image index response is requried by API to include the image annotations, which gives Ratify to oprrotunity to perform some basic filtration before invoking the verifier on the listed artifacts. The exact mechanism for requiring the filtration should be specific to each verifier as the behavior and logic differs based on is actually being attested, as such, it will be part of the verifier configuration. -# Filtration Methods - -## Age Based Filtration +## Latest N artifacts only verification Assuming artifacts are generated by ORAS, they all have a `org.opencontainers.image.created` annotation, that markes the creation date of the artifact. Based on it, Ratify could filter out stale artifacts and only evaluate the latest image. To achieve this, Ratify would have to read all the referrers, ordering them based on the artifact age, and only pass the latest one to the corresponding verifier. This kind of filtering strategy is best used on artifacts that are rapidly changing, for example Vulnerability Assessment artifacts that are immedietly oudated once a new artifact is pushed to the registry. +* An artifact without the creation annotation is considered to be the oldest. +* The annotation value should be a date time string in RFC 3339 format, any other value will result is invalid, and should be treated as the oldest artifact. + ## User experiences This section describes the experience that users interact with Ratify using the proposed solution. In summary, the propsed solution suggest we should allow for filtering of artifact based on annotations, and as such the following section describes how the customer would configure the filtration. @@ -79,7 +80,7 @@ metadata: spec: name: # REQUIRED: [string], the unique type of the verifier (notation, cosign) artifactType: # REQUIRED: [string], comma seperated list, artifact type this verifier handles - artifactAgeFilter: # Optional: [string], one of the following 'latest' to verify only the last artifact, or all to verify all artifacts. defaults to all. + verifyLastNArtifacts: # Optional: [int], denote the number of attached artfacts that should be verified. only the Last n will be verified. if not defined, all artifacts will be verified. address: # OPTIONAL: [string], Plugin path, defaults to value of env "RATIFY_CONFIG" or "~/.ratify/plugins" version: # OPTIONAL: [string], Version of the external plugin, defaults to 1.0.0. On ratify initialization, the specified version will be validated against the supported plugin version. source: @@ -87,14 +88,14 @@ spec: parameters: # OPTIONAL: [object] Parameters specific to this verifier ``` -### Considerations -To implmenet age based filteration, Ratify has to be aware of all the attached artifacts of a givan kind before handing them to the verifier that wishes to attest only the latest artifact. In order to implement such behavior some modification has to be made to the executor of Ratify and the verifier implementation. +### Implementation Considerations +To implmenet "Last N" verification only, Ratify has to be aware of all the attached artifacts of a givan kind before handing them to the verifier that wishes to attest only the latest artifact. In order to implement such behavior some modification has to be made to the executor of Ratify and the verifier implementation. Below are two proposals which are currently being considered for implemetation. | Approach | Pros | Cons| Notes | | -------- | ---- | ----| ----- | | 1. Obtain and store all the referrer list
2. Sort it in descending order.
3. Use the CanVerify method of the referrer to make sure a verifier
that only wants the latest artifact is invoked once. | Naive implementation.

Does not make a huge change in the executor, other than fetching the list before hand.

Transparent change for verifiers that do not wish to use verify the latest image only. | The referrer list can be of any arbitrary size, therefore fetching the entire list may cause Ratify to hit a hard memory limit and crash.
To implement the feature with this kind of behavior, Ratify would have to limit the number of attached artifacts it supports to some constant number which will be determined during the implementation.

Additional latency for sorting the artifacts. | A test index list, with ~1000 artifacts within and two annotations (created timestamp, and another text field) weighs around ~400K, default ratify installation has 512MB of ram, so we're well within the limits of 'normal' use. -| 1. Split verifiers into two groups, those which require only latest artifact, and those which operate on all artifacts.
2. For verifiers that work on all artifacts, no change will be made.
3. For verifiers that require only the last artifact, the executor will manage a map between the verifier and an artifact descriptor that is the "current candidate" for being the latest.
4. As we iterate all the referrer, the cadndiate is constantly being update, if a new artifact is discovered.
Once the executor had finished iterating over the referrer list, it would execute all the referrer that required the latest artifact against the "current candidate" which is promised to be latest artifact. | Does not modify Ratify's current scalability.| Requires the executor to be aware of verifier type, possibly by an interface change on the verifier API

Requires changes in multiple places in the executor, performing the verifier loop another time for the second list of verifiers that only require latest artifact. | The benefit of not pulling all the referrers, from the standpoint of keeping the same 'memory footprint' is not clear. +| 1. Split verifiers into two groups, those which require only latest artifact, and those which operate on all artifacts.
2. For verifiers that work on all artifacts, no change will be made.
3. For verifiers that require only the last N artifacts, the executor will manage a map between the verifier and an artifact descriptor list that is the "current candidates" for being the latest.
4. As we iterate all the referrer, the cadndiate list is constantly being updated, if a new artifact is discovered.
Once the executor had finished iterating over the referrer list, it would execute all the verifiers that required the latest N artifact against the "current candidate" list for each verifier, which are promised to be latest artifacts. | Does not modify Ratify's current scalability.| Requires the executor to be aware of verifier type, possibly by an interface change on the verifier API

Requires changes in multiple places in the executor, performing the verifier loop another time for the second list of verifiers that only require latest artifact. | The benefit of not pulling all the referrers, from the standpoint of keeping the same 'memory footprint' is not clear. # References