From ecb3a7cabf9805c3a480f0972a5be34eb5b40375 Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Tue, 15 Nov 2022 17:32:41 -0500 Subject: [PATCH 01/19] Initial KEP for improving pruning in kubectl apply --- .../3659-kubectl-apply-prune/README.md | 896 ++++++++++++++++++ .../initial-apply.png | Bin 0 -> 24198 bytes .../sig-cli/3659-kubectl-apply-prune/kep.yaml | 40 + .../subsequent-apply.png | Bin 0 -> 43791 bytes 4 files changed, 936 insertions(+) create mode 100644 keps/sig-cli/3659-kubectl-apply-prune/README.md create mode 100644 keps/sig-cli/3659-kubectl-apply-prune/initial-apply.png create mode 100644 keps/sig-cli/3659-kubectl-apply-prune/kep.yaml create mode 100644 keps/sig-cli/3659-kubectl-apply-prune/subsequent-apply.png diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md new file mode 100644 index 00000000000..563a8e1045d --- /dev/null +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -0,0 +1,896 @@ + +# KEP-3659: kubectl apply --prune redesign and graduation strategy + + + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Background](#background) + - [Use case](#use-case) + - [Feature history](#feature-history) + - [Current implementation](#current-implementation) + - [Problems with the current implementation](#problems-with-the-current-implementation) + - [Correctness: object leakage](#correctness-object-leakage) + - [Scalability](#scalability) + - [UX: easy to trigger inadvertent over-selection](#ux-easy-to-trigger-inadvertent-over-selection) + - [UX: flag changes affect correctness](#ux-flag-changes-affect-correctness) + - [UX: difficult to use with custom resources](#ux-difficult-to-use-with-custom-resources) + - [Sustainability: incompatibility with server-side apply](#sustainability-incompatibility-with-server-side-apply) +- [Proposal](#proposal) + - [User Stories (Optional)](#user-stories-optional) + - [Story 1](#story-1) + - [Story 2](#story-2) + - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Test Plan](#test-plan) + - [Prerequisite testing updates](#prerequisite-testing-updates) + - [Unit tests](#unit-tests) + - [Integration tests](#integration-tests) + - [e2e tests](#e2e-tests) + - [Graduation Criteria](#graduation-criteria) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability-1) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed (Optional)](#infrastructure-needed-optional) + + +## Release Signoff Checklist + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) + - [ ] e2e Tests for all Beta API Operations (endpoints) + - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) + - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free +- [ ] (R) Graduation criteria is in place + - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) +- [ ] (R) Production readiness review completed +- [ ] (R) Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +When creating objects with `kubectl apply`, it is frequently desired to make changes to the config that remove objects and then re-apply and have those objects deleted. Since Kubernetes v1.5, an alpha-stage `--prune` flag exists to support this workflow: it deletes objects previously applied that no longer exist in the source config. However, the current implementation has fundamental design flaws that limit its performance and lead to surprising behaviours. This KEP proposes a safer and more performant implementation for this feature, along with a plan that will enable it to progress out of alpha while continuing to satisfy the needs of the users who have come to depend on it over the past 20+ releases. + + +## Motivation + +### Goals + +- MUST use a pruning set identification algorithm that remains accurate regardless of what has changed between the previous and current sets +- MUST use a pruning set identification algorithm that scales to thousands of resources across hundreds of types +- MUST natively support custom resources +- MUST provide a way to accurately preview which objects will be deleted +- MUST support namespaced and non-namespaced resources; SHOULD support them within the same operation + + +### Non-Goals + +- MUST NOT formalize the grouping of objects under management (i.e. it is a just a set of objects, not an "application" or other high-level construct) or require the user to do so to use the feature +- MUST NOT require server-side API changes +- MUST NOT require third-party CRDs to be installed +- MAY still have limited performance when used to manage thousands of resources of hundreds of types in a single operation (MUST NOT be expected to overcome performance limitations of issuing many individual deletion requests, for example) + +## Background + +### Use case + +The pruning feature enables kubectl to automatically clean up previously applied objects that have been removed from the current configuration set. + +Adding the `--prune` flag to kubectl apply adds a deletion step after objects are applied, removing all objects that were previously applied AND are not currently being applied: `{objects to prune (delete)} = {previously applied objects} - {currently applied objects}`. + +In the illustration below, we initially apply a configuration set containing two objects: Object A and Object B. Then, we remove Object A from our configuration and add Object C. When we re-apply our configuration with pruning enabled, we expect Object A to be deleted (pruned), Object B to be updated, and Object C to be created. This basic use case works as expected today. + + + + +### Feature history + +The `--prune` flag (and dependent `--prune-whitelist` and `--all` flags) were added to `kubectl apply` back in [Kubernetes v1.5](https://github.com/kubernetes/kubernetes/commit/56a22f925f6f1fd774ad1ae9e04bcf8d75bbde63). Twenty releases later, this feature is still in alpha, as documented in `kubectl apply -h` (though interestingly not on the flag doc string itself, or during usage): + +
+Relevant portion of `kubectl apply -h` + +```shell +$ kubectl version --client --short +Client Version: v1.25.2 + +$ kubectl apply -h +Apply a configuration to a resource by file name or stdin. The resource name must be specified. This resource will be +created if it doesn't exist yet. To use 'apply', always create the resource initially with either 'apply' or 'create +--save-config'. + + JSON and YAML formats are accepted. + + Alpha Disclaimer: the --prune functionality is not yet complete. Do not use unless you are aware of what the current +state is. See https://issues.k8s.io/34274. + +Examples: + # Note: --prune is still in Alpha + # Apply the configuration in manifest.yaml that matches label app=nginx and delete all other resources that are not in +the file and match label app=nginx + kubectl apply --prune -f manifest.yaml -l app=nginx + + # Apply the configuration in manifest.yaml and delete all the other config maps that are not in the file + kubectl apply --prune -f manifest.yaml --all --prune-whitelist=core/v1/ConfigMap + +Options: + --all=false: + Select all resources in the namespace of the specified resource types. + --prune=false: + Automatically delete resource objects, that do not appear in the configs and are created by either apply or + create --save-config. Should be used with either -l or --all. + --prune-whitelist=[]: + Overwrite the default whitelist with for --prune + -l, --selector='': + Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2). Matching + objects must satisfy all of the specified label constraints. +``` +
+ +The reason for this stagnation is that the implementation has fundamental limitations that limit performance and cause unexpected behaviours. + +Acknowledging that pruning could not be progressed out of alpha in its current form, SIG CLI created a proof of concept for an alternative implmentation in the [cli-utils](https://github.com/kubernetes-sigs/cli-utils) repo in 2019 (initially [moved over](https://github.com/kubernetes-sigs/cli-utils/pull/1) from [cli-experimental#13](https://github.com/kubernetes-sigs/cli-experimental/pull/13)). This implementation was proposed in [KEP 810](https://github.com/kubernetes/enhancements/pull/810/files), which did not reach consensus and was ultimately closed. In the subsequent three years, work continued on the proof of concept, and other ecosystem tools (notably `kpt live apply`) have been using it successfully while the canoncial implementation in k/k has continued to stagnate. + +### Current implementation + +The implementation of this feature is not as simple as the illustration above might suggest at first glance. The core of the reason is that the previously applied set is not specifically encoded anywhere by the previous apply operation, and therefore that set needs to be dynamically discovered. + +Several different factors are used to select the set of objects to be pruned: + +1. **GVK allowlist**: A user-provided ( via `--prune-whitelist` until v1.26, `--prune-allowlist` in v1.26+) or defaulted list of GVK strings identifying which resources kubectl will consider for pruning. The default list is hardcoded. [[code](https://github.com/kubernetes/kubernetes/blob/e39a0af5ce0a836b30bd3cce237778fb4557f0cb/staging/src/k8s.io/kubectl/pkg/util/prune/prune.go#L28-L50)] +1. **namespace** (for namespaced resources): `kubectl` keeps track of which namespaces it has "visited" during the apply operation and considers both them and the objects they contain for pruning. [[code](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kubectl/pkg/cmd/apply/prune.go#L78)] +1. **the `kubectl.kubernetes.io/last-applied-configuration` annotation**: kubectl uses this as the signal that the object was created with `apply` as opposed to by another kubectl command or entity. Only objects created by apply are considered for pruning. [[code](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kubectl/pkg/cmd/apply/prune.go#L117-L120)] +1. **labels**: pruning forces users to specify either `--all` or `-l/--selector`, and in the latter case, the query selecting resources for pruning will be constrained by the provided labels (note that this flag also constrains the resources applied in the main operation) [[code](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kubectl/pkg/cmd/apply/prune.go#L99)] + +For a more detailed walkthrough of the implementation along with examples, please see [kubectl apply/prune: implementation and limitations](https://docs.google.com/document/d/1y747qL3GYMDDYHR8SqJ-6iWjQEdCJwK_n0z_KHcwa3I/edit#) by @seans3. + +### Problems with the current implementation + +#### Correctness: object leakage + + If an object is supposed to be pruned, but it is not, then it is leaked. This situation occurs when the set of previously applied objects selected is incomplete. There are two main ways this can happen: + - **GVK allowlist mismatch**: the allowlist is hardcoded (either by kubectl or by the user) and as such it is not tied in any way to the list of kinds we actually need to manage to prune effectively. For example, the default allowlist will never prune PDBs, regardless of whether current or previous operations created them. + - **namespace mismatch**: the namespace list is constructed dynamically from the _current_ set of objects, which causes object leakage when the current operation touches fewer namespaces than the previous one did. For example, if the initial operation touched namespaces A and B, and the second touched only B, nothing in namespace A will be pruned. + + TODO: link issues + +#### UX: flag changes affect correctness + +If the user changes the `--prune-allowlist` or `--selector` flags used with the apply command, this may radically change the scoping of the pruning operation, causing it over- or under-select resources. For example, if they add a new label to all their resources and adjust the `--selector` accordingly, this will have the side-effect of leaking ALL resources that should have been deleted during the operation (nothing will be pruned). On the contrary, if `--prune-allowlist` is expanded to include additional types or `--selector` is made more general, any objects that have been manually applied by other actors in the system may automatically get scoped in. + +TODO: link issues + +#### Scalability + +To discover the set of resources to be pruned, kubectl makes a LIST query to every GVR on the allowlist, for every namespace (if applicable): `GVR(namespaced)*Ns + GVR(global)`. For example, with the default list and one target namespace, this is 14 requests; with the default list and two namespaces, it jumps to 26. An obvious fix for some of the correctness issues described would be to get the full list of GVRs from discovery and query ALL of them, ensuring all previous objects are discovered. Indeed some tools do this, and pass the resulting list to kubectl's allowlist. But this strategy is clearly not performant, and many of the additional queries are wasted, as the GVRs in question are extremely unlikely to have resources managed via kubectl. + +A related issue is that the identifier of ownership for pruning is the last-applied annotation, which is not something that can be queried on. This means we cannot avoid retrieving irrelevant resources in the LIST requests we make. + +TODO: link issues + +#### UX: easy to trigger inadvertent over-selection + +The default allowlist, in addition to being incomplete, is unintuitive. Notably, it includes the cluster-scoped Namespace and PersistentVolume resources, and will prune these resources even if the `--namespace` flag is used. Given that Namespace deletion cascades to all the contents of the namespaces, this is particularly catastropic. + +Because every `apply` operation uses the same identity for the purposes of pruning (i.e. has the same last-applied annotation), it is easy to make a small change to the scoping of the command that will inadvertantly cover resources managed by other operations, with potentially disasterous effects. + +TODO: link issues + +#### UX: difficult to use with custom resources + +Because the default allowlist is hard-coded in the kubectl codebase, it inherently does not include any custom resources. Users who want to prune custom resources necessarily need to specify their own allowlist and keep it up to date. + +TODO: link issues + +#### Sustainability: incompatibility with server-side apply + +While it is not disabled, pruning does not work correctly with server-side apply today. If the objects being managed were created with server-side apply, or were migrated to server-side apply using a custom field manager, they will never be pruned. If they were created with client-side apply and migrated to server-side using the default field manager, they will be pruned as needed. The worst case is that the managed set includes objects in multiple of these states, leading to inconsistent behaviour. + +One solution to this would be to use the presence of the current field manager as the indicator of eligibility for pruning. However, field managers cannot be queried on any more than annotations can, so are not a great for an identifier we want to select on. It can also be considered problematic that the default state for server-side applied objects includes at least two field managers, which are then all taken to be object owners for the purposes of pruning, regardless of their intent to use this power. In other words, we end up introducing the possibilty of multiple owners without the possiblity of conflict detection. + +## Proposal + + + +### User Stories (Optional) + + + +#### Story 1 + +#### Story 2 + +### Notes/Constraints/Caveats (Optional) + + + +### Risks and Mitigations + + + +## Design Details + + + +### Test Plan + + + +[ ] I/we understand the owners of the involved components may require updates to +existing tests to make this code solid enough prior to committing the changes necessary +to implement this enhancement. + +##### Prerequisite testing updates + + + +##### Unit tests + + + + + +- ``: `` - `` + +##### Integration tests + + + +- : + +##### e2e tests + + + +- : + +### Graduation Criteria + + + +### Upgrade / Downgrade Strategy + + + +### Version Skew Strategy + + + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + + + +###### How can this feature be enabled / disabled in a live cluster? + + + +- [ ] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: + - Components depending on the feature gate: +- [ ] Other + - Describe the mechanism: + - Will enabling / disabling the feature require downtime of the control + plane? + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? (Do not assume `Dynamic Kubelet Config` feature is enabled). + +###### Does enabling the feature change any default behavior? + + + +###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? + + + +###### What happens if we reenable the feature if it was previously rolled back? + +###### Are there any tests for feature enablement/disablement? + + + +### Rollout, Upgrade and Rollback Planning + + + +###### How can a rollout or rollback fail? Can it impact already running workloads? + + + +###### What specific metrics should inform a rollback? + + + +###### Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested? + + + +###### Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.? + + + +### Monitoring Requirements + + + +###### How can an operator determine if the feature is in use by workloads? + + + +###### How can someone using this feature know that it is working for their instance? + + + +- [ ] Events + - Event Reason: +- [ ] API .status + - Condition name: + - Other field: +- [ ] Other (treat as last resort) + - Details: + +###### What are the reasonable SLOs (Service Level Objectives) for the enhancement? + + + +###### What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service? + + + +- [ ] Metrics + - Metric name: + - [Optional] Aggregation method: + - Components exposing the metric: +- [ ] Other (treat as last resort) + - Details: + +###### Are there any missing metrics that would be useful to have to improve observability of this feature? + + + +### Dependencies + + + +###### Does this feature depend on any specific services running in the cluster? + + + +### Scalability + + + +###### Will enabling / using this feature result in any new API calls? + + + +###### Will enabling / using this feature result in introducing new API types? + + + +###### Will enabling / using this feature result in any new calls to the cloud provider? + + + +###### Will enabling / using this feature result in increasing size or count of the existing API objects? + + + +###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? + + + +###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? + + + +### Troubleshooting + + + +###### How does this feature react if the API server and/or etcd is unavailable? + +###### What are other known failure modes? + + + +###### What steps should be taken if SLOs are not being met to determine the problem? + +## Implementation History + + + +## Drawbacks + + + +## Alternatives + + + +## Infrastructure Needed (Optional) + + diff --git a/keps/sig-cli/3659-kubectl-apply-prune/initial-apply.png b/keps/sig-cli/3659-kubectl-apply-prune/initial-apply.png new file mode 100644 index 0000000000000000000000000000000000000000..43469c5bf9cb0cd59fcf81ee51e20e8595bc238c GIT binary patch literal 24198 zcmeFZX*iVaA3t0ZQ52Od*|`hZk~IvvD=lIQW9&r8PFZJcQ+J4v?7OTnjO@c0qcjxR zmtn@f@B6+!*VMh}|MGeDe~#z(!f{-n$$p#nFwJPtkx+d zr-I%pIXyWaakI67UgWui`blQdODS2O^84oO2^WMR>!8&>Jzyc z?>)NSpB<$S#w2nFtct~)kSd)?WOq2`*}<%D*PUW$w2DX`z9S%i?XM?EdWvbe&Ot*4 z>?YG(Vma=k&xu{j--f3iZr-`R)b3t$vgL4s-Pl0hT%w&Jo^kZcNyD3Ft}PSqH{T$r zIq!J=al>F-@_Wuz6P@d)L?3vm^%0)K9) z#bX_6RI}3J%d||;8ieL5$XsRjq?07U^}}Jw%+-qgru&%prLdSbKl$=CnE} zy!9pH`5+z~?fEJfM&1X|ul|BOoEfJcgkLdYq`iEa&qyUVl}Fm;e7w(Qg-^`k_L(1+ z8_hzyiQwHt?`9w3YT{h4R(IYyI^xD^LaC)?&n~CXeJWZ;9u>c1$3Z_IQtwbJb*GJM ziv0TZpTEC5MQgavcl1R+6*W3qLxW0BrqF-B{rNH#+K1-%7Bi`-(4zf{r|VzEJK$2<++=u~?~HiP?MDF6!HJ=nyj$n==L z%JrOIu_*1*L20{Ij7O{Cw+P}_2i1Q{0kQ~C3X9-|FhFKGL6Vj~?z#87!qjvX_m*mf zURDicygDl{Xb*g4-35isyA+dl@+&6dlhDK;ar;L8#VtyRXqgbt`AZmPU;Lo&;l8Cz zY1Ju8%PQBkd&G}ZL{FP44dy)jnngo-x3{He2wRleBrxDH8sgF@TK*%=sn;SVQ}@c> zI@ENL5hFu{EdVOODh0aB;e49}OsHeEMOpb-L!SvXZtT6W>!d+qN%KA zpu61OF3^x}14mp*lMGqANnx|^1*&@Y&%gT=2KppwIZ(d6_LVA2BS}A}?c?M0BHs33 z@oEo!_;jZ5s$ax45|7AzXg7mE_OhaUo;XVoP`OK zgTrU7R|Rhs_S>$8W#5G)$k;~@=FFFo(E}sySYzkGLNiAn<(0R=Ec7;wDFveul$|Es zOC9aE{WZ!0Wo1eEn76H)_4Zgm4tDa>JjditKDoW9 zt>~8fJ0is1ti-W#DhWz{o<7a&d zk2ikSpQ%|ME^kkzhh^X0=ew?Td67f8${t*k@!na>O}(-(oanvLy9(Xi zW4rliIj1W8@ffb|lI_!5uSOB{uNLwg&Seljb)S*x;nA5TYVZ1VH^0nQm+ddPiU|pj zS|9)O3t;Oa%mQ6}KJbw&zWjB{I?1*B`viDv=#|h*yqBj}or1dpJV|(-$f;3c@2vY` zPmJ(s^4lSjhU*>!cqyI1X^os01K!rbw;^|(mVL&w63hZ^ z?he89#Jj_IlZ`S*-n9)xrWbop?taWSfz`U$&N7dSoGou!ZB*tT{OIn$m?B|VUaalN zCQmL)_ADKz-E7wE%+Vb{ccc2ty05hGlgf&R2ZwdY3le!4ls0+6^T6l~`AiF`(xnSs z*rJHv%HzDs(P;kB-8l(OlIRxA;y&D}hmzN&o>pPR_$C54@_J70Q#YIXBg2^EB4xa{7CLW~JS^Tn>eo=?)0mbSr7D=h_bm1LY%o zmKycf`bKWW$%Bf;xLGfP(+S$UJBcGj$yw?B>GleK0`jBZL#wK%wkARl`L2wadKczK*SFU@~srlO^n zK#qngpvjl+m=*()t(|}2mXd+couwEC`;g~^{nT?_>ydnZLB||;w5aR|Ql2YgBgEY4 zP~-2pi!mlXo?V**UT@qMMV>XS23iDgJim7!y{kB#5s_Q%zJ6KErSq&W+DMip@RDqn zi+lITu)D)$S?EXM!sQsl_W3Kz3R)aG##rxib_2jg90%P3qAmH11lTxWncsrWDR#bq zOg<3K?`-c%c+q^Ln5&9*&aK~-xYp$8Q@pSe?d)jZuQ2Z*YjI2pWfL3BajaGl2PoHH50i;DACBS^_4 zba${@FpoT>?ZhR9hcjw3_QnMG?E-|I_)mX}WCWr{FvS`LpAxB@N)y#dn_v-upAoO# z8Y$Ubz*-eFzgavHW%H_;#M_$`h1w_uJ1<&0zQPR{2C^!7c|av**scs{EgAiA3cWEQ z=%Wq+uWM6wDTmXtzpHf=Q(o#}vJ8k^M?$}D48?l;WBiE2^vfr8z(919QZv<-PGj<6 zR>YYqLxNt`*mrfQmLZ$P#H;YCZjNL^RzCA*OR6M-ls&10Ng13uB_w{*c==t{lr>s* zJ?k!fJPL-ECEMZ57;M1o=uclxLklu;tP;@OZw7`vEaN?yY;~&YbJ^h%sU6vFJ%RTyqAu%v7H2r(zyda@yBh z@u7+<-?#$Uh!HTz>{omQKPKeEdIiN|t`@UL9Vr**r*q12 zLTYQ*x)V&6yHs(#cdHkGh$0apGhW>G_*iLezAg6`?b(v;i3Hh9uIra6y7gH&wcd59 zEb09P9+lK_>bfv)lXZB6pc8#gNR8COct|p)(7b*_t>7s{w{%li^;*T(^J;4)dR;5i z9oew;tMytgqe3;{OfOb@$n1FIpx0`X*Y({z++?ZE6YlCCi8qREPy>Cw2BpIRCT6TI z&VsIMsw@GISr&yu*Q>Cll6O`qf;#{R?uLVJsQ@5H9aN!mBZ0Z8(xA^;8aFZ_H}Cf$ zNOJ-|x(CcF4i9#-Z!v&NU`i5HZhaW%DEpB&Ri(9N!gRJ*W{kvTP?Jv{66R98R=$rA zxw1zOMvJuiisSLzvY>U$BgW%4^WKDke*LnlJxY8eMMZy+>z7qwe0BpV*({fy`%r=& zGmu>3KH!;cLsm_;F^GE1Vq&vc__oU?g{hB_1uae&xM4i1`Ox!|P9m|_#Sv~~i?}_S z+`6FYxDx6Wbm08?gDgYnOfJXr*D13Gai+ZqS-eoQ2GA>En2r*0c?kmP80u#_rhe`t zF1s|DxA^h9DP(YR!@)z!3Lhg8xdu4hdOqY%Ei;YNzC}E+O7V7spq@7J7qHqSWP44y z9^g4S#^;f zArF1CF)XFAHiKY&z_zU=BNp^ZAFR1`pR9uldu?4#&Sk%efCwdP8_wO#cYkJY2ZQVk zGx+M3lH_OewAOLq;&+H2Z&7`hxGfJUz3HnoDDX3e{3mZjYEUQp>`ZG+5*3QwzS&U; z0s3|^&wIoULYAsj1Z9)B22=-lFBSO4Q9`0Zn-inCuV0Jr&J{z-amA0j&06f}M=9N| z`E8asY_SZQv)k)BLOefQGScw3jl2PI=606oI=N4<$fBeNsW>3*nD0NZfdEn*q%toI z;p+)a!_+g>I>w&kx%zh>Ek_l2WnLGDkHA-bx;$Pw z<)zT*w~ewC#Pq^v^NV4_V{FQ+KP0Ey)x?|dmMP~H>sosirAfO9lO4TQ%RX(o#&rs} zM|rg=KB^1loR(dZ9Ki7!Atze=jgPs3+$a8Bp6m)~C=E+LwTn z+h&3=ift54Y;!u*9CwY9HNg+Gd@JuM+jcAc?DNl_(Ne&O6BNPtM#aY%lrjPixD&5< zOtg8{K^_j`_sGXN7RuZcoj8XpDO4O_*Qa+tSMSVd2INibzBT?~hBITe#X*#sSEptq z7r>$$6RO^6*~KVhg2&8CVLg|D5QORzf$RH-58tQ7=0c2v5+`_yz|}>oqd=!?zOP&U zgZGOk+lZu3eOgj5!{XHr97eo@;=-=EP?S>MKIaPX+cVuEcl@r=%uLtSZ7owoaRe=B zlr!yIo=XhhyEmrO6_d%0-65&v9$|L$ALi{mY~4M(i$y>qW~AqhqQNHY@e6Z4?Lanz zYgA0HSMaBGALnNGAEjn7Y@JP&E(=j9$u?f?*<2@@43?xyH=MH$18;1Zj@0@#x)e0@ z{0v}OuY+m_7Vq84lDYab!OEn){f@O(D>x_l37W8Oo1HEo3zrQ(zO*!4TZ|AxyhyRM zuYR?@3j~p|L5m(CeiE{w$7BW^CA5?^G6S(H1bo7Nn zI2FoZn~ltyUF7MhY4rU5ym~(zt2_%BWWSmrmTTG`OXVLfQ6%04$=V!G^Vq^7ndXYc zq=C-1zEBG77!a?vW?d8_@i~TN&1WRMZBahlYiOU+`JAjTy(?QULowD*OyQk=9 zJ$Scw=VA8Jl%{`98u$m`GWkqDE}2>jjsJ1v&ngM0X${FG!_b8P-UJeYJ3xr2tN-A} zua*7PMr#(}?quvex{fwmV|tDeb2d6!Q{(53{M~X%4zMxm?8c=>zrUtSMTLHqnHl*r zo24uuWpwse0ZgyMA7+2_dwS;M$l3pY4|QwHROlHtX0;1uq7JBr7ks{yNJ{#1|KAlQ z-7`)0oVcH^8uDPZS;3LLOCM?rxRz4B6HEIhhdZ#!^)_!Qsq$qqM^*~$rNQ#%+Sl;mOtr4BfCv} z_t$bD-nr7JQB0Y1^awnER5FSl!Q`>@nWh66ViJJa#4l8>HggfVCCr+oyqYW8?OxC? zIvWZ9(*xkkuYwkk_~^)4?WtW+VU*>i{Bx*l0b=I4)xOmmxr+Hna@( z7Y`2{D*?QuD7lO5fE5KVf4u-j7(e;ZBSq|61#p}s_4~?&~<7O)xqN7J`8TLL&vEso&_^WPB!))&s@{5Z(%Q}P8{%@$^L`g(6+bb1Ca~($i#cg8ZinAc@AFH~KxA zET$WRoA0+AjON&_0~oB#eEFb{HGrzlTEA#CPyt%jwo}ZhEPlJGNlbw~|;|?vu zwx!?AHV*^K*aIlxej#Fa>NQ~hSab0$PnEmLS>Gv;M00{bEm#LuX2C$kPH)%MfXdetH-C@FPe8>+|+L>|zw9U%ELDI9_ zWr!hk;g*;T?|)*D8bx(hd1`jLJjN*YqBVeNLaY*EB%UwY%4u_9hB59&OMLq7!*XosY^Zv#ea{rlHDLUyGLhcmI((jJnTz zFSf6{D~Bs#w=}0IMzK45JwSQy!^NQPXp7cSl6o;&IM$2)*4+gaVe+BPde2hh^tY50 za#K9>-;h2r2^Mc=*|tmbLl`Q54Uw1uP+=$>kkO8e zkn($X6J6Xc^jaKM2~%ddg(b)AU>{fbB?+_J6da?2WLp4^MRlxLuwdMz+75Sp55>I? zR#PBhiYxc@t>-W<6PJnQUbR9z|BAqI&$eVUT#O!pAC&-=~ zQ-{z8@|agoJ}3B&+gh)Us7XT%n*aNRmYxlHM;UuNh?P(CE7LiHdrjw_?bfWvnup$} zz(o4br^nkh5c7i^78U3J)4ywfQ1kf*>hhTd1&U@&(3ZFk?aURoWSQ*Gz~bh*Biz-v z4>1zcP218D!v9VX`9X+L;`Z0`;mv>;GD-MrSF0G%pqB?SV~pWKAqFp!N~X&q8i`T- z|9%0$f5tp!ZvusYs9>QVt!^`4cE)h@I)@|h@!W=xBT`d#J}{QgEZ9Ru>+2=W3XuIa7ZCuoB}2HC zmArQ(i!(dKs;-DucNYDhpIz7@n{cZm?D6;Ei~a8HfRBmR0Ys!s@|n(Sa#8(y#h7}d|E9*ZM* zsN$$n>4402jQCNDl(WLJT=!yOvfHiKDTsv%ia*HkoHXFsUudq6pBMH$BGaUYrnSy$ zSws_WX+-%f1vvxy(QqZB>s@C$bK=H=P2o zui09c`BB~HN;KkblInV=VgR%|;9(92(v^0r7~QKO`YtZFoMev1=_*#Bbmj7=`Ywen z7K<)8;K)P4fKX+8x&%5Df7s4?rLS@D{BPbs7}?@0XSf9@-F%mu&VpfTK8U#Y`5`JK zrM1Gp;)sMW3f?qbckOO@?wn$KDwb2Pq-oY^z-M<}3c$yo+H5L;(>cHuNBq7@Djl*g zIOJP&6|g_n8@*PGW}?Nj=bj~S5m}5qcvoYs2DWFN3(p>1oJ;4wnTIJdTMHJ;3}WU% z#B(p2`X|)UJ4c4G7NEj~pfzMCIwaJr11{)C$wWLNxvp#MUk3*~Ai1bkdicZcz;*ja z9H>ZFoRXBh;dR%Dioi+2>aOXzBYj>zHCA|+D>yRJ-oUa$2004^eUZ{jeXFMo$t$u( zng532v}?sc$76-}lox&O&7L`rUEi}9Q=Fuf<8+Hhp%t??kwx$t_ve7%Oxk8{gT z8Jc*GN9Em_NH7_am-CU1djGE`)eJHJoG@EoDO*o_x%$u!un!jPtIhb~_tVqH{*_>qbm-^|0qqGMdgm)u)*b5uZ@jqMb5ewEBfei6cEz(Mmyq;6U3r z?uc#y31~VHt*KJm1Mt?7m(?k!+flHIBO3;!?;n9E>G4hS@84?yNsRh<)X_i==!%jh zYPW9u{=GI3oNIYn{5-?@_dg*(Ld$-!1bn3T{|!~;|NE=Dw5%%m$s0l2=Q%4nU9I3~ z8`h9}`8dd%S=V&zi}fq@^7|=q_GLQy2@`uWqdwwd)v?66+ z+o#nG?1&7AB0A6se&BG9b@OI9kygJ&VKw6>%z^rfO6nmsv%W`L32E!pK7RCONj<*L z9VWlOl|r9(Uic~ijNAl(5mx>CY`cI!&Z7(y9NcTF70Rre0Q>}*Oyz)I6`w{UxF1aK zCwG@9varg!y)K!s5uPyd26QnM0;*4R@kuEwJ_j5u1KmH&@8Yun@7&!jxp$xG)iI>b zd0p=jhkWhCn_A5Eoi{tQPd`Yt0MtR0EOlBZ%`ZSQDo{xHh>p@S3Yq}fS9K1@f98mi z^Zt%Y{qh~i6`-NxCqf1wK)047lO^gq`Uidp?dO7H_SFZ=t5ojT8lthjui_1 z&u4LZ+Z3B--T7_1oiBX?lFp)cEx&3+6Q^{g&Ut5Gm%E_Teg70nCu>*!v7%0-YvO{f zp1uV(?LNf9tUKb}I6%$<+_CK)FM&tdHD_c8qcr5^M_fil-J6RDMw?2JO$rkktHmw( zvMI3#wgDj>Cuc(vEc%-*6P=s8rh6F5ja;L~JCbBmx_WJJ@#TZO^_kd3gdExO(A)i? z@@|_Fj#M3`Q$1D0-T^C%?~LTLFRT4LU09!zJBI7BoXs3ed~tgmdkRDq&(C&IID@mCd~Lq8kvYn>G7?rSw*6t^&Y zTQao6TU0HbOnBV{q*?hy7CqZd1F0ON?yk*q{1f+Py_e~%i^W46NzOQwcLD@B{WuSt zemIrJw3qp*7*-6LHVXCLe`Iyj*6L2xT0=t7x;eJTcqTz5u?MZ>TwYR(4pf(7B$@jc?)2v^?nhjoF?b#Or-=>F*%6-lL|ud z9Z9R|hy~2#@YBT@)4MBOOtrmU|H@GOV^;Wr>Y=1+-+ajhmSEJ<0;&Qr6y*hqM_N>71eyw`%V$a=B z!)FI;^Nt#5Dzt>MM)WN1ZpAKG?}48R?QN&?w{EX89WFL2E|b9r%E5;L%P+nbPG;1e zRTdpooPRMSVZDE9w36OKSU6fLJSIJ3j&wc2W65+cZ)H2P!{0#tyJ`j1*T({h;!gIB zgS1gsT5$jt54bK70Tj~lUziwKL6hs0({+ZmeDDkX`7}mmC?b{$h1GT)< zVrys`c``+Xsz9RQGQ_#?nnP1K5SKgx!VhNYhZm;)C8@UXT5(u-DHJ{g>`r3Q0%8SmI8c?b=Ho#b@x;VP!bH8<{WU>CZJZq%;aI2=3lgm z>}>wr<#r+xmZ%3l^@-1K8WnsBGH>TqmMa}aLKvW6bwyvWnpJ&zgNFPSvp<>(#4C6c zO6S^xGffiL=0`^KvPN0C3U)o$3Lq^-_{+-W_ef7`r8~;;r5z$4t zc*B4=DCTfu6T15b!Oec3?4b0)W>vTVS2}~RGT{q+3%m36`+fUXmcdc)!ey6l3-C)^ zo^v^&*&ND|`+Mo5Z|aRQ<>G>g{9hFquq2ZoVdI6yV4;RmkNP(jRn=)_d)ph%1qO*; z-Dgvp8;y43B|NoE3wE-8_S6>ZRY0 zM2C2QoRdc`>VPxlnh6&MTI4qya(S+o1s4&P1w$*&jk!jR$i;I9jL3Bv1$nIy_dB^7 z&3waKJ}ApeBl=-raMt4kj7Ryt*xvyh_O$sZ);B84fBoTq%AJVn{XS9rqXAO10+U~JdA~lU z)nXyQ?Wx+Pa1KI)_U_l9`YS{!Urxi{#w3ie z4(0s5Voo!?gO3w3;x!eH&)i+`t!hWH;oA5oV)NViISN>V);6$*dGIr;&K{=xunGQ; zVkbKkd|e|F<#*pMN<5U)%!4^Hc$Oi3X&Gbh3`LU{Q1L7b6V_7p`1Oq3aFKHy?!;hr zmiCpQ8=-emzQeDE>zu3byVU{3BVSjpC+isQj*ZbaL~r$d>2wpDEzN zh9Njt>K+K%Ez5CwYT(Av4H0wvErPX|@jK2s1s(znx#e^8*bT*iL2FS$j1DRN?zM3A*?bNLRFc(QnmGrEACW=Qy*>mRVpqFQ2tfAWi z#Og-ejU`03cCruqtjtQu2pF%|&tG|1t>7jvRkkJ7f_X65#YmE4dn4dIA*9ixabv=- z;ZdKL&)PM+%Kgt5BWgqC_|^3{>3Tm6du<4{R%3fYH9(5-Yed02SR@Qu8qk2~g-#at4P= zE4V0*2@Q;LC-0K@M+_cu!U>L&V=hh&k~eBScG$T)Z6OD!FXf*I$Qlgf{(9bm*^ko zRJhk`3uJ9$0L0Xk5ujcx2C_ric1?KTvDc7-%TcnPbr|_BjVRc(b^! zP#KSbiI~id%1&}Tp4JPofSI@(I7j)!h1OuODHw2u2i<+v{pB6>-K90P5A%NkWf1`U zRA$#9s4Gl+`x^0WCpo68-WDXU(EAJ!nJ zt6oj-u+rPFv*rd%d!U-SXQh^!WrK!FJ(R8KzuUV7U8pr{A#jS=v_Gik~v6!5?zz?y} zTJGWU+*teZ+IzDv>-`;+iA!*RnfKv4BkaBbEaxo{zmhZos4cn)-M)TWYNs^?y&-9~ zWoINBZ{Y(L5ODsYRV6UBu*;QnIVIWh>%{{_`@UtDM!2Rku0DC|Zr7F4&Cr7}1VTr2 zajtQ?o;xv(W$gQ|xLR0WZ#j(GHQEujx+3thZbdFUWT>K;X?2wc9F4fAW%Xd<+($^N z$L<|(gEgOg3^yLDO+Qtszv$=f_E9A0SV=j{^#a_%M#Dy1abMSRW|3z7?PWeaLv2*Q z6)>Uty#=vnzz!lMLUY{v_fTti$wf4Zp0Khu>C!3d8>>z48chiSgon6@`sW5uF;OX% z_x$v}5S2tU8V+_1rkUxA7?ERR5nkQ34Q7Gx z$2KCI_zOg5z5E{ga%w4{OsJarGchdN$WVMhDMLh%fRg)WD}G_6t4Irw z(KS>$^L5+gxM3DwF&v6=b}px)(3*uxg&J$!rwy^~*HcuyL8UBJGi00YQp+EC3m@2( z>a8q|az5WEt@leV67HVw357)yAdXs!$Qulbi`y?=I2cXa$H9wzj5pwAy%V+pI?2V1 zZp%(N7|72u3^5+3rpsREr~XX!SjT!$@^-%P!dpumMk;?YvzyQOq)?xKlnZgbnRcD&8>b8jU@E2w<5rpH0 zULP8tJ^C3!;C1)zKYT2(SJ@!|-;h=&pL<4wjMC9f%Z)cOTFTYW2@uDz5!!F*K!Y1A zb@jaN+H{@0v~~Vgz-&cG2GVG4H_Lf(`7G9FiL1ZIQ_-k8n}&J=!8`x%81nc9fbqnN z<78k)Hpv=e&tp{M*m30&FSAA>?8*n>n&nKT0#A+d4Hf#zWjD>#S@93Lu9aOM2-oMx zh0%9nulo8n<5x%q!q{N0^^pigVYf`q?`yv zizp431Pj+b5tc+qoI7r~T&m+doR0-obzmUbO{+?mt^PY)YyJR_s3FRGU)05I6X2!k zHfB>xk7iwse7|6>z9$9eLg~DDRZ`~x29`Z>>}sC@(j%~5iH1YVQ&+VrjYp$u2cEh6 zjyq8w`j@?I%ABb zrcE`{!&VUg{gAs3ai9Ih#8=!qg84%TgZ`{|{gj3GUID}c8~owNx*}sPwy5i5ZGXOj zA#?r5YhjZ-dCE_^M$)1ae2Pb7uTq$+!p940rytaLfvAyYVTOr3>+2S4G;WG}*>bmC zM=D1v=h*`JI>`px0p!JkNRJQH6FCowl8?y>28-YVXKt8ctKiF~obtXTKEWSa_KPa> zTFR^IK`DjJl%UQYGhQvk#R&aTfHCIpU~ylIW$ti@Q1ErPbxs;_nSbp+BJ3J83lR8H zZIMa%%KKekz*!u6>$m}NXZnT{Lp@hiAANmINY(@|z5z;=V&Mo0$Di@c1aZ*iu0>_` z8@a4H2#1#KVwC8B{_|}|-$5;%-W5zwB@jY3g`!GbbA?~|7*5w&cR|v{L;I%d&3Y~# z2Z>USLfZTv=XIYC?g)+RYQxQ3=A!k7J>)gApY%J78 z*jj)w8!&PzX+hTOG%$bGHQ};!9tj`J9Xlr=kMN#NN3>q0JADCob8cF2h}ljY%-@{K+nA$71OrHM$oTuMQsN(;!@ zOVGHl5QgCzBVT3-u$t6#3u-94fOlE#a9raw<+?JoZ)l6J@3J=Xl$`gwl5Xh>kLY zJfgG7#Cz+gsOW~5ZCm_agC~5|HxI|(P;f<9(y=@0>DeC!F68MdYdn(agFmPoVo}bO z4}WtZiKM(?mAp0IlR*@kfmJPvXz-05kXD*crzz!~$}(CVp+f@d$jy-y5C|FKcYJBr?9_WJh zy68;gUaZ_Yzvw7>!1eXjlQdEN^^8x!HnHlaD%2x~jTVy!Qud`C<-!b#!8MmdFJKo+ zJcF_s>TxU1c0q?Gn*)a)^%Cm^)7{G8cOyLp51oHtrsisci{Xi(!C{WL{RNTwbWE$R z7g&dbWfu2M7u!MGJep>aF$FdD`k_GJaF^)*eT>#%iHbU!8B-v=i z67GE2OZv3D-M1>9-4Uwrac1j+?<)#`^MHr7eySEcI@qhB!y|@%cu=OD{wUz!6pl=&r4Huu_}%DpVpUFQ4g> zVI8fjpKTN0e*J`oYr3s0^@BCXv>by_W1Fstp9rU#0E99|^=&MF2xsv__>&z}@m^ur zg*=k*+WLY(cQ9%@>|mWciYDmmwVuoJ8YQYBhCJh&P1RfNwos%;=k!ja#?;Cm0`TX- z#u9m3u@U0;)O;MmrFQ31k~6u&2fJ9fpswecEl~L|s_x|rLSllQ=9g~Gt}qEydV|?Q z6h}U2crrMbsP`!tK-gCHcmGh~vA8+?syrp7o8;8e;4EMmgfxd}IP)&C2z2>=7CTtH z8$=@jWYIo#Ax69jEcI1pL9U~i9wHOVqM7b{A46^cU@QhlGO$-;3{*aFhbSn_CcRbuw4Nn*F7Q}E`RAt7 zjiSl+F^j?)UTNhYC6W(Z4)!@RJV5{7?*)JtPvZ!Nm;j?EVEdlayQpo@Fog{e$ z-Ui|u3tz-gQSR=a>1V@X_TZ5}3b8zxGQ(}9gXO-v77deKKHb&+Q?odX$D;(8SAq9O zt$CbYlD#?CaZvH;vHGN11~SU#*?gX@PRpB70<{(A04i1bd0@a5Aa&_LB^$is9t=VR z`pJa`Jg!#NYZ?Vl_agY8p z;bv+G)kt#d)|=7W2fkHX&0Vy16jXteUu8m#X0G^FL8#|)L&4tR;qykXrJ-`#2cyQ_ zWO;ust8{&N{~vYB_UqyzDczj-=P}@9Z|nC{;1Ku7y20R{LAnkT)E@3z7}8nfa6?vr zxL=;OA-6v^n9`^U8V80Ob7=pDW}Z+eqMWsn8Cs~m>gC83brdaqWKF%l?^3AoWW|XtQ2(O)oEd-7MbaO0GhFG&O#joeepXt)9cezR2ACGx zbsH+Fm#GFk7gIzZpMCuw^Q_eXK&l-Hc@R^^(_e8I0*&9-{o6n5%-4*Db`lmo+Hq|5 zKcEmSS`CjgSlOTp`cBcu)_^y`(PFb6nD)To;!HZRBzX$(|jAk-~SfL`6=Nv2Iymdnr`@eqYLCOaASbxQ%H4H9(7 zKVflZEvbK*X%L?BX#nju15g8z*MSlnv0IP1sPrH47(X&e7+Du(N%HoXw;QA!ey7qP zl@G@Yiu1zp2zWHLCFML*Wg0mAP!ovaMRqf89q?ka3>>iH!Vjwq9HUyq`%#-*WvZf) zl;V$k-cR!r+*Kpi69$V}!!Nw9P>#5)U#-`bst*5r!Ja}8ex<5I*kB60a8&?0-@9R> z?x(U1N-?n0mUlTUh)OB^c?O><22^{@nic(hOTpjVBl<|c8rRgZ`JaWLzmDU9iO!s7 z#B>v${zjZ+WEBKlfy#HX_;2);&dWSb`N7|R?!n(}0@Tt+0MiEfzTD)ut^TM6FalU2 zC{@w_zNuoG3Is|gE<57?Wyw$;9>oAsA$CRh2x}B5K2ZVgnW1?w;PX4BfWm_}dZ_bv zsmrg%kw{<$XCtfjkMfNCRE&UORmSH16oh~KI|0mqQ_yqwKiVEe7%G8bxk~80`Hhos z@&SPCJzCjN`;9^53JmLi3;X~3g{`Td)o*-2rNkr$92{%y-TP^kFg293WF;A|t~$y0 zCw^+qKL?*NP#hszw=yqp?3P~*BeO2Rz3r%?6OsDMR#&4-Rjf!?-}H0%_|q$t!Z{b( zyla*sG(ds8DR3>5^*!QJZNnLWAD2+MIn6fz$jJ3b;Z!O$Rh^g&pYWZ8zl_u`5JyQ; z`fR`R*w_FyxXv{G_q8g2Yoj`V9ms^})cva$KlRWb&_nt=o<}v7Qcw)=;L@Xhr`$yX zR|<$yjNnoAZj}O*v*Ve?!`~Ff0w^bGd=yFZn<0@>2XMsy4fTJk`u|>5RmfCmI+0H* z6YK54EiuDa7>>Fh7vW!Y)cFQxH{n)*HVfRzk#w_4ai8FDQ{;&H_^Gh{;=)>t^`bpX z)Q9AOqB9Hoy%{y}5?ce5>=B{e7yQ*^7?$ff)&^g*K0bX5eGVGt|y6KCJEd3_2nLUQ^ofeMR<+q6ENZfkco|+eRy58fmmIK1(+%1 zUN|xb(JQ~^U`AlqCRhV)c7A-USNkJzxh~a)=k*bvXzFx&tT2F!SVt}ZXwCqdC=A~1 z9J~}0@yn|HJ)KI%t@^=~d*=|h^*x4PV|5$n;M73>XWju999fF_?$m~?@90wQwffgg z;FvM<6SKwkz||W`#*VhH^r%T!o%(^wJ4qzsz-2b?{KQe?lllh|d*GoK-EorL?vZBb ze3ClAFsD?TbdwmAogj$vO41SeY1KvwaCyr$`gW+x+RylqF0YoIHoP0gl~nhZEh0o3 zzuL`!_A61lh5A)?|7#BW{7btz#4Ml!Ojr~hUA}o`74ii5kCpJRdf>?^xwph z$cr9ZNJKHnuB1)%guy%BlFcWJ?isw=DV4eBZ~3QqPQ5- zI%$<~f3*GEIT5tMHBJ1mWu9!O8FXBP)DPuR4e3O^jB#e6WnA} z{^N2LgOliOY@|cQL2l)Tyq}k^{5=zZE1K26M7E0pc5I_0cO1v~tg*&*GRa*-IKM=* zakb$Cnb%DC(F-C6FUYB3&{?QtZw7oYE)ReRJSm2eg5GR?V zA)3l#o-olB-E_c8MfoR) zKxv?odc2enZfQ?)dQTEEch?PjWJIPPY4>)1L=;h~L;@u;R;WqxPG_1Oy++yNW=dm> z-;E@x{T4dUTnuC#iC6PY%1^FmUU#s??NF1HehO{}vw>Z>a{b=(3JzwH|DwokKYWOKIM@JNIUl%|&VyBJX_cDS_Y zqbf`F!Bl8Y(V`8KZ8lJMC}5i~8n&eYA88BF&rgaBI3i)p=RKxhPj-C2>#Z!@29!O- z5ST=@t9W6NKLrhjQqq}HGuhFsT&SFFr<99M8hWa6zbIroqMWykOb`33-#cK({Y(1Y z$eF=nmCruH+-qjD>q}dyh=woJ^)2Hp+S7e2lsQW#VRUK_j|#=m38Yr&&Q0bLGfTr{ zR7`46sy8pm*y&;;*3|aTTwM5TJ4hIP&%mL=qHJvJ1z=LU^~)nle;nCSN`dXcSMzCM zc62y&8@QU|zIlD9efA8)Gb3uW(Gs_7=i}*jx6Xc#9@t%bE7pDeNS!&-?Hnt%ConG; zX5-aya;?9D|Xk{_f1cT4giJP;>q%d4W zu2un!#8?NQ{<5U?g<^j3$VZW9 z6e~)df1>9jQ0cftxj)FI&$-^hcA=^(!-en~pKC6RI$~R|=~DZ^ry^aZe6W^;LH9v# zttxq~OL9QghU`05UAWVjoq%&mHT@e`UtoXi)7s8#!O#2kfSavELb$+{7lX3sIRP0w>2rq91H4$ZTEy}H?AZG4 zjGT|eerzLaZzf5O|cZryDjpkuG<#SZR!1w5(g zTH}?wTI{kV0anlF6kh`FTb#Cb+efi@&)e^ReckdS=yciknW^wF0uHAxIB@^puZ&$E z*34g?y?%#=_EDS9XD0Wq`g8Mqoms=uO|RDho7S->za;&e>~A;I&*EXrn(trte!u6v z`Mja~t%}ocFZba>VOGj~rbX9juxLzeko;875p8NU7beEz(c?uG{^w%jbUUMl;2#w6g* z<^JpI-XZVg`K$85{{QFqCI2MVm#o&aGv90Sw>4)y@cfo!u?@#;*9K(+2To>Ru-ksG zDjT?sck(6C@R*5D_x+f3ddbG@HFJ-xHC0@@?OTi5o4{#EK~bVsaDQ(7n+mTFy)WP0 zUbb|r}UCi^FKwY`+&<3;MMRc!5yEk&E6b->(1BXnJ;fH{~vd<#0Kf6ob%Whw09$#;}`U$8=GXYliCUt*)m=r!bvgOsP)qH19 zZFgMDvhMGl6Z(6<1cA1SZgwzZx^>euV{*WCi;LEs?>)&;HGpI-tdvZaCFbQepw z)toiGzJw#`@l-2N=K2LZ*x>!X)HI}>9JwkVPWsmwar-q+zPwqAAc0za-*dBAx`>?U*5OVVj7}FW9!1&YN z)@6jM6nHX7)ItN)UhP`o;ZV6p!vfV1@dq?u1+W9EEdoA+1>P}TAPU@tvi647DTJFr za-u-1S`V=xm2|){)kVNwv1qjGOn9OLF%~ERN5TSTAh+v*`(}Vicw5qH4df;_==4>V zb)sCz=P>{WFuQ<38kQrPid2IE$Dz7_Ryiet&w+v478MBWDtE12xm)ece8-0{?ZG)RRZo zqGi8yU*T}$RmGv?ypfjOD$+Qk^6U%K39eDa8?Q~HN z$(m&PP!v}@mO$lNG7SX7dx+A=X*A8m6y=*bC&w(t^XyTIp~U-q*VnVgH0G;xTzSxW zx3Q7$pXXLo4(0fE2B9rD1D8uQX2xQcyn`f3r1Nb!CbEw`2bQ7^U-^P*<^1l78ZYv9 z<*}LT@!@kS`6*9-a`Q<#dL@#Ok1F>}7WSGLBii*b?=UfB+Wcqr;Z5En-Uv~<%tstr z(wEEav3}%Jvn(Q~T$o(CU zT(`wn3k9;2ImLOx|H<>8%ZfF&+M+(itZpgHiU!gCb&Oe7koBC%c22lH@14SEERSY)Z7!Ly*^9BYOIKl*eq%diIAIoF1U|{|GJ_rLN)E)!- z=V#P_SMG==|;ycX`>o!WLIWdXHR5lk1-taXH9@D*5w`y1UgT?uyS% z2{c!bb@9`>Y;V5EDc)jt>$5h!-EDL_7C57gT=mW7UrrXSM%FeoMRb<%+xGWDn~=-N zOTB(er{`)#x@LTLu}NhxF#q?j*LO?J-c_~R{HJrIJ77G8?{fd)DpHnw49vXuZsolH zdpz_wAT2-K|1{ofHW``x+-avP|3?Z^nV=wY zhtvE|%@xV25nk})opzFQHqVXmuQ&a6TIGsn>~m6VbL6HkHTI}gHwICY8l8p--IbmlDxek`1(X7;c}~fJIT5y4!m8t zO)R=VDxF|g55l46gZ&y@rv?aD1jtq$l7EGC!)`iWqTvrXJ4(MtW$N)xbiJ_r?2wE} zg@J+M^MdWS?Bm2r=P4nVL$IS`jeOd!Pb^U&p&3GYp4V z=*jG*cZeNx%-H$`g*ErYZDZzvw7p}?VzP7mpn43pu0B5N;C#LydJ8UAqc4tlZH0q|ty6UFRH#ra`eSp$%xuQkH&o*n#_I z372VIp*C^7b1XAL%Cq|2+UY+M>f)OMP}l6MKhjHjF!oB>a>{1pk4VGfiAiJx8Ir}N zwPHTkKg54x#0CHUvY{nNbTu8jBI<9m7QkX* zM&}<@eb`c{uH);~UtXMstm=^62<`FuG&Jm138VSob*J?hgd zGGc5XY5or4SHzV%>{o1mL%Gptv}zxyn#jlKMst#CN6iUm%aa@K7vE0aJ^g2Ye?fsW zo~v1B>Kp&htj?ke(?xcjS8Mv+t zUn#|Jx2yGlLPjg{j;YVt_s$Tx_ra=bWp+cDxHm1BQ6~;*v$p)If=^DAxf`v3Fz8IB z`gC)m+_vrXUgPmb`M}w^RdXl-jt^(y*>-4CpjV*7snJrhd;cqr*9XzTdO|BX;Ckrx z)^ZkbmK~C6q=tN_a^SKcfBehel+@FMjeIpJe$1Iot7Y@{A{ZJqs)=eWzjCRL=o@6L z7LPMk_s&%|7O_u-H&gj3)Fn@WdzV(hH=?-K!S&bb`sHFL2u-l}wJzo|%f_9v){AqC zt-K5ytH@o0*hU-0eBAuD0LBS3`j*RX0*F0Trod^zRk!kf=ka=je!Pfhk8)g&(dQb( zhR%`*0&}xtdh0c$hv~HT^aRL1xGs$?N(&g%qiWVFdIt>A31n*oxuec=kl;J=VG8i* z?)N7Xuvy(={s9U8l0{`>JQo-cl{;M!#!}_QE5%ni(-YHS_Z59I#k+a36ZJLq`jA0jq0rLR}ju}nhjTXF4~ zhD}|+q|07tY07d!HFk7hjm&9&g98suuWMK0C~0TL(6f_fQ5q#~(O6TC-L=LnJNtsr zpRl@07t;X5pa9{I9^U?!=ZE-jSiU569SS~TocReJWV-MHY+q}75B4KM8gP3X_xxd;U90-Z#S`0IZjsOP1wMl!&0-UgcG9jth+%mEuoXukh;A= zWkfZ#`amK-^R92aD-*}dpJ`K&3$V)$C?@zv7`HIRtOVg#A81yWigbnNky8k~j*d6g zYrQJo5tS0Z;R(pACghvqJ%Q6c9Y#L;_2~nz(v1(6olMx7y(ikD7?pRUND_Z(rw^!n@(ly4!vB>^ zMhnxkx?DCUlh8^;+mdaOpCML}hpqkn@tX-sMb}Zqs*Nt@w41;^6Wr=ORg*<{4LmrVARMF{#Z5ksy#mg)o;!H@%p-|sM ziHD1*LU>JujS_Na>Z#v|?^?3$9MD2+))*Oe`ad=nAvs-;dHbEb+SpX2#O-PGlexC{ zl-lYyJciKVs@^xdUTWx<{m!7p@x9da`6caQ%gOWJ6k_OnxAhLBGN3FZ9zb#P7e|H!DifJ9+B#IS ziN>Lghci>fZH{!{HqLzkN*%6K@d&v)>_4467ZbMcU;%f!_OmF}!n;B%CNMWNA1=-{ z=U*qh23f*+1H_tG2n89j5TkJ`?T}JY)%##^BYo5F!(yzaK5Kb7!Eyq`I$ZX%EwI+L zs;N|vw$Q4*pXO@qaw?&e(nt}vLGDLpK5AQ5a|GkOMh#NAn%z2IVh`bjY-$u;Ne>A- za6Pj3&l_F*osUh?`MBPNh5Fyf6H72&Y?YlWpxknqfS_d$`i z0}5dgAx@`P=2nMQ!+hGHftt2RtJAW%Mm_}9Mv{T4>4BPL{F0tPmugq;M+SvWW8*0- zE_F`+ShxjU85n5FQ-%6)V}ifn42?@mXZnPZ$vaYK@>FL)g=74*>e8$=m%m@+&h<1! z7)iCD)?jQyB7ub*xGiQG9eNA%DfMkry=``+rtsh1v5p2k_EKT3Y*G^BiXk!q!1HLa znO6y9VkC)^Z9L<+SeutU-~SsO9Ey93RqgzY%Q zBr@Q1Q?}*dsV?lnCYk>h&WqQYQqK2fWra!ewu73K*>XI-p4&5$)&;2^Opqt?J=p{&0?q1EqS1x=rtA=1xx72 z3+LVRAu!iT;(2s$M-u?*A**h14MP5UdR;GYih%mkWc*~wBs8gHQ9kUt<{jPP!&d*v ze>iiR?y%)t2yEiWQkeL!X2rXmTx#5jwa)nIxumQzGHg*VNPfcJ;-J(Q%h?b_U9F3y zdK!?;@Yie9la-HJd^6toLnGb5_|uu`;h=|*$d+>n2}G{u9TWiTm?{Pdmh%`w;<_H{ zOF~Z{2}7rELq)~UyV*;QqS@4$IGZ(K;s%57Ts^q3j3j#S)(aM{+W=twNzoBJ-c!sDq7syRF0j(p#$7yYW6hFiDqlL`@!~5mdL3x zLTUf}%aJL8i2u=CC_b{jsCs6SB{yl~G^zNe-;Z1YU@LAk382)R5JUPifbFuTyG9Gd zMXDZ)(ALZb6ZY=RwV!iP>^(_Kx2@I=(+7_@$tal)O>>2Q8kd^-Ly#$O1NHr$JjbtE z+WIb!*T#5yh2XwGZ90a^Zyo^eE(HTaY|Qh-U-3H}v=)`iOu&{$X!X?M(5;ZnTK*U0 zX43>pT*fWkwI9{LOe`J?hIMq^EkH{nO@cRiCAMnxTf6@nn_MSIwvVt|dFj{S3b2fD zv2G5d3cD25T+iXiTlYs32d?~#-9YvlBu#&=W50o`^h<0*JOCFkuNndPHQiH3i`wS) zF9IX$mJDIu;5bPP=wD3%gkvuFUFx&=fni{27zW}vjhFb>KQ|-^*gv-EOQP^AK6IcS z$#@uHK}Yse8Z?$=*68u=Mdc5&Q!@Up{StS=arj)}{x^+0jIPm#W42b`q% zde0Az=9er#23)w?a$QUG`v<-s(CyD%fFl_bAO*O7GNAjptUp8CznW!W2G0HWy1!NO zU%mYQ!d_te?ndxf<$JR|+|2?2XUS(CcwTvV2*8T#=l-Xg)fY$r+1diGY7@t84iQlO z<7Ypqm~8B|kpkPb;XF%#y>k&jq|0JmxH*RKGcA;i&?<2H9vIA1U2Kkz35G775a-~`;FEKH)`g?vjH;72I+q& z=(|}N|J7H^8V{&&Le7uAz2Q!x=C?@zhzs~JY8f6Ob;9z`Fp#tPoQ*1Y_8airvE&lh zM+J%2Mg)l($5b2aP)ugS-a~?c`wnAj2JymZ*Js4%xHz&z%{-)s)9V%UNEhy0=oJe{ z`vkJ@x*tU$bF~T{yHu2xcd0%s_hapx5gq< zWy0D_W4-8NjCNSCZszV)1_s^7rw+hU+DzBDE&)Z)UZ=m-_cfoQRIf=dnxTAe_wvxO zYx&Vs;F*=ci=2;2YOX*;fVC0$UCu`=jU2lqOb}8fc)yw9b!ogzJzko!#-}MVE@`r_ zLl7@j(?INmrtK`t=U&hx5(yV?s=F*SLMlE8yJU#j%umw5iG+@}pLzbFSQ#{K!UFRa zPa>ofDA*gMw{N;+NuOsN+dmiHye#68KCfuwNR}QeJlP+6cF+04giXM9dZ2GnZqsgu zn%P!SN#^(5aFK6#s7~pNuQ#90!tmF3vIT&yTkPimYDu38?^zgb@80TA$hFEzduI2K zn5S?uV$qLITSOc3XdBPhv=|KrFSXh-C)%3%bV;v1A-eGt%S>qMpXU>m&f-ImG%S^# zQQsPR{c2Y>cri5XvtERc*cCe2bWvSyJQV8m*m5FXFLfb` z3s*%aH9_{BorNx&vzX=HVSO`vg0hJv;{=;N>uNqrMEZW#JF^iiAC2$hTiMKqgdI&za5yA zMG{7p>N}|t@Hp7}_?Xou>+qv>7V>fn8USsIkUs4V3@jbPSOB7o5Utpk?it*?xj;Mx z3VU2X0HQ7dm{|Y-{d@G56D>4PHuo&+;jo!#40PcyQ%SJm+H}7oj2Ei@boi{sMj~j` zc?upLN@qz|olK8gV{#Ph4L`rv)=A+#42oyMn3dD%YjNZUnaDe!#V zd?^(00o*Aki+-_nj?0q<0Q4I=!E|>4qL0lyigovE(4sKHBigXS?p^cn+>Fd3R)fqi zS6@<=!J7!Qk9Jc&-I=iM`;1F}rEbYn^vuqE;X@#g@0PEOwSMv>T4z55A(xv?d_@BM zpI(0}Dma8RyHoOM*w(S`yWnB!<3J!H@$PtQ9z586OB$)P6%}8LTIxe(6st=eXQm;& zjGe7d4n@B1N$h@ZSWss|D?yZi!=Pb&`sua-Y9T4-HKhJLd?_O5a>DVd=RHxqP`*DP zEvBYr#rQ>1XlFQIPn^Hw=w+WzcIp`4+PX{pahBA#mv?$8^i3Xl0U6+0iQFPZM34^b zU^RwSTLszV-xo?Bu;MZolMx{lzqjfv^u2EF9eUOd2FO~HI@OIfem-E$*w=}0DwCg~eyi5&z zeFvx=G5Se-Of}RbzCVy&rkoPy4>>vDH$8AL58`}>W&}0pKoM#w4v@x6OjJiwagX!l zwxycrykB{z)M@-fFw0NY5Um_JJc)x@19#-g)undE?c@Et?aH_^VdYRF8XKU3(zv?M z5voKQ*^M$AOkX@dJv?;ZY&e`*DxosB5<0}uxPxA`5jGy1Zx7sCzdQ=qr-4)@ivBZ3 zy|fUptiz7G|GOw4&3y#3Mg~}JH5`laUo_R9M>!o$A(LXI&ll9w{0}CU;*24AbJz4b zm+%!9U8T)EM|3X|G`qD-_aF^HpB?DDy9h9`yESG@4iUD+H+ToRk1y# zp?rn-a(~@Q02ZH?_88JQM}WGh-#)TbK3~2J*e@8bbkyhnTA!!HRtPiNK=?`>XMAtEMGus1tcz07>v}@4V^-nuxJ8 zEYK@r2+e~cosEQ%yiW(P(aZf7)R`*XqT;(q(Y)6F@Vswh?6}myD!u=Yyw*%l7LU(+ zhsfxSa)t{Ua^#|Sxl02s$_rC(!iAh~=p3B0MysArR)!q@TbGTnd@1f!TtbjY@z6Mx z!l+M=P}vjV;`(>6JpZVf#F;*`mz_vIXZmdi+ftTzx7$(Ft|L1FuD1MkJ$ zSHAoc@^8<(yLtcPJa$Gt%2B51s!YpOyTK39^kI|V4&U}wj7i_UzBtbF>X#>=4fwpj ze)vkgrOCZV%L8dHKIjH$gm5h(V2f>~Kv#SzE8c8(tifePqx@4g+~yr&0UD~s-YIRL zdze5mli>!PsjT$=mh~K+^bJH~Gti;n=lHserGfG7eM2P;u^P+ujkkQ{!%fcjR-TbO zjjjX?Bf%pCmqf_2GkkxnZl^+ODboM&RU+0K^a8~)mTq@3!%LcuJDln!mX>B-4Y3ut z4Wo_bClqA^3r#h=8(X^!c~*D0jYO&>@Z5St=Tb&xp@F*YMIv>SF8ci=*}0&a$cBdd zRJtthsEsBXLhWLQqi}fA)`%>MDK)v9YAbfx6yE-ori?7UwjF0LYvSI$-kZSNz5&pi z_^793=0TKN)Jc7C;ejjHd0Na*NL|iYf;FkzK3oUk=@hoq3SZd|(afKvJ(Sk{u%G$( zj}AuIzcegDmR`Io@KrQ?F6DocK+=GF9O*^|l=(1ZcPx9Kp>2wF9G9uX!LIyR^Y)8d z)Pu#eY^H8ym3@n`i$K(r*HgURk)7RcG=h8)Fa zfMZ^Y(N^P?zC3f+tzR(m25C{_r9x-CZI*fyZI(CTiS@;Gd?A7c*nlQ;O<Ox}( z-tLlFR@+qDx|Hz(H+JQ|Kk6Xe!B$%vLf#DAU0p&ppFLh1&9gLnItNBAOJ7pMZ(?7+ z&qg!@^qL@bUYibUqeYG#NRjD{CRD%u=&U7L-ELqTP6G=cGq*hHm&d%4YP?jY?%Iob zaY=vD5CiG8>*L!bOUJD0aH3IjM+uB|E+YQI&UnB;Oh*d64}vs}0(QM{e;P7FG9} zMuZ)*Pt6aP=VuHK|Kh>X3F|IMe7mY%^dKS!oW|mnejUV`iXN?`H3SzxHlVy!cC+!8P-Po2|#sJ@G7ci#g6AaKdr;m@F%ppiQ}t%F%M z3s@v46jGDE`7^}g+gg&bIoW^w0WXtOWH!BBb!`5^BH(;qKalm)HF9kRYbGF?F9=wD zCPe|%HQWqQmw4x3o%AH88tn?6_*bYUJ2=k25tjR-D69t1NNUju&IW=&m;dtwD^Ec)cFRy(}4YWV~qW&J^ zjbrv#cDK*mhbgQAg-#J~(t@XQqNsbe{fF~|rO)Sw>fD{z(5eFysw4{`N=)nhiFxWP zC(45zl}7%IL;BfNX1P}ks^_y=ITrT#O6D-&BBMtVN3&}=3bfraOL@(_en6+0f_vwJ zIW57yq-kvo)zV(4=jWft@OxqypkvY}11Zj7x9E@c9O$q!8zk3>Z30ZD8-9UDST>&-&r%)CcvacoG-weI!%osqs$Y#>QkNdD+@K z9<$fQA$iXt}p6k;RM0fw^ z)$%RGHcb~N>+t(^bmoHl+3)|l^TPev+2YPb#Ei8NDrtN{S9{;a??>}mH^fYZ3^*8#5t>C#6gN%~fmeD!$<(sFF3 zRJj!-BS;`2Z5u9~a7+x8ef^=(0(sCMEDH?X4-$)?X*yn;!HH&4?G|lT>PrV^;BYTz zI9u+G5~6dM5Un2tiCJnyd3qvxPY$ng~V0q_;5i}?t#&R^Uq%mnOZ_W_>vDJLz!K?>L-XHs79 zdA~r6VSl%W`&7JAuH1&A);*6LZ3%fTIpzWOGiD8943)n1jr^u>7ypO?#GF?$J$jD5 zv}fnrf$C-0<@sU^*0@xTjoR)__&ZxZJvHI?j1a0E)axus*}F<_OTa-<1uq(dbxVt+KB=lSo< z93*o960b6mEIr1h_p`VF=3<+s`MlV$5;G6Ib*RBUVYBM?j9QS*D0QJBsK!EeI8`zxet+on~6@c@F81_f>&5uFNjy3p5Ffc-T zKG@6*#WHClzG1Z{IGbQFVvfTW@mr!uVVKb7*v!T`P~Gmvxnx=C0I~o*m`P;lzT;S4 zjK88J`aXr&HoYn9C1Y*g*A(=qKU@P=>{p1z!vH5Q0M3gY!ygkLn=Z;zC(Lu0-27|a>3ASui2=@(>XGAO+IPm(IiY972dLdWQ3Q5`4TrG5%_D18bR2mD}94=4uoOY<3Kamgx?E z(rk#8%0_XawFCHX{=?aI!4US*B6Xg8p98VKaz9I4(B|_UfF~w*jJ7UM0d61LNB2G* z?SYu!rW3DyEINm=V&j&gNWs5MWQ6sLg(_j*TA}}%5m}_tW_u3$0y#oI)cIQHz-V^8 zncTK5+aFic;0F=i*uA^X0^n3lzF%F&KWh&hvAhI8QE+J#>tDC|Ws@>7z&RG`r9UOz zA7dqd0I)~023_R;W3-<@NeMU?a|QKRAph-b1h;{*pP|s|*Y>Jk_d+`oDVl*%dU5>s zd%wH`jAm3x_4xj>pI^zE^d-QmQ82dq{bicJLLMv_O>N_`C;o1v{WbUlfGB(&^+@VJ zO*VHl@E2*>9z30_WaaCHx5*za|IR zDJg*bz>gTU|I-{;fOG#{zW=jI|6RWSdf)$yKL7Q;{|3+hX%QH(|4#+a(g0aE#-T9m z8MCsFeitM9)SNx{fZ!hh@&&f%2+cfc-(?qJ{KHJ5a|!_mkT6?X^2)ZMBf3j{P^uiG zX9EBn4FhwDoeS{q#AtWd;E?}STceYD7?8NzYHVT(h~z6B>y_nu%1HsrHln3hl(6@8 zEPGE*3x(bwec(|mN3w8zn}eJusZ0xjz>EXUkkxwj0cVLE<IRu`0oV&yMq7UFwu2=qvyHm_5^`IizC-Dd>S{7$?zzN93%IIcN0ytDr>RAn=prp`CzJaqSn1M$&US&A1rbNaM& z{7br{h^FbB$<;T7Ba04*;U%U7I_2x{V z%!tK#8R1T*!Qk+@@6xd&%(soh_`rX#cgAe5Njh*T8EmsR{wiBJ=U^$BUyTQsew=G{ zcdPd>jzh5Bg_^3j`=N|O5>;58t7@ummAzpVCaWXz!+m%d2iC_c%r7Ep2EqcJ9NGq% zZw^fe+7l2;u# zmhmKb0$$gybWfed*%2r#inERjZax=XxAHwjL?kH>Y*+UC*#PC?Bk-`%-n-6hIF(rd zg2TM2&GA)T3o?BQw7+uaAc8iz$$1YOHP=~ST(QJnBe}DC6|-ZtG5qxin~=eWHYSs3 z58VRuk}CVg_3&)x^`&VHh0ssScngXayA_5uZDa#*VPpW3M@+?2#yv`?f4{=m7%E@=U)+f(%Vg?opN(H+hF#)1>m zJL0ieJO#HSVL))Z5@~R^DJWPo^rmmWSke1BsN@%?MbD7+1+e15xIeS>{DeDFNLk8c ztvd6>W0k74gRAjG=o0ta0UW2}1V+M_`|VZqiHY8qNokR}S_`BQ($%&daGfX z7bvu1nccCkfjwm1S7$Ys-n|-N3@BYnX}o1&oGxVF@Vs_qI&~>*<{`7Gl!W_q$Uu@| z3Fn?|$?ip8=Awg0#Ex&ZZb7h#Yn^>4D!=QnjDFFRnm#VH_*=1JvKZ@ZE*yi|Yx|n5 zU>rbyTmeJqQMrKuYs7+&^mTgzY0k#p>ZDZDO%;`kE@rm#i|Gw^BbAr}8!xgy%W9-4akjRrIxlr>qO`o^YQbG)T-6B9>}JKSB@ zSq@6m*2)6rOr^fmN9{C@oiK0LGP~V zFVl$+bVFC~{M_@i_os2I3FrALUb0|5k{e&x$8E(P4P7?@9>iy92r}Y!M%MX$u)jno}2+@NWi8AIq*&!}hi&My5S`K2cAsaBWu^3kxCCo5sgSwJa|VdEI;8 z;aYiAvu3=_G*p1D_w0xJC*?0LT2>O27IDSQ$)-#us-0F7bwt-4SK+>{D%sV##V4D7 z1*~?7A`R@7tX?5lTEF~O%n46KG;)~&8{EFDFO7iqzuTG#45*sZaExBg+01n9mq(ms zF+kH0Zn~3u`!h8!y^>q+4Htdi78Y(sJ|2oOoYHgjx@F4Yyz0r>_0W+M&Rl}0Qy?=O zNmf_uV;A43qFs^|2uCEHN)ROEvLhGl`9n~8{uai^(jEhXBi#v}jTB$lb+mTOI+cZ2 z6MP$laq@j!SSFDDwJr^o@(4-ZN6iQHJj3<)#&EqnE`}kz1r~rZc_^c&PX5J|Xv)tp zE%%LCb}I`!I%z4K-nTLEu&saHx1SXC6&^8B-7uQoLKJvxTo#n0tev=%Y6?b}^=G=F6 ziVb>RKN&01HZN=9%QT(Y47JU-kJE5%&3!C~yOBhN3sL%6MZU1HvJoPA`(|RgVd335 zxyh2XYyO*W50iRukaSP>nk{MqQ4OS~_#tn3U@{bF;PpPmG%VlDzdgZ%U+fFg74^LJ zRD?|IbAffN(L>g}+6tVtaDKHHTTfJdUBjVe7a?g6*G-{3J_IIpnQy8NrH^{e3j+|7 z)YWuf4m^YxLn8D_rtF)mY^Ji`mJ#>n-W-sWX8x=W)}o!PnB=ujh14R%nwnClsjW@YG`Oke-xrc7DcD1tW+KDWN5oj zsztH_8|oq%p7ehbWj=~7cHfb?`!={L{DmZ_)I53leYI8xSmzL zIejyuO}Y6=+de*d@Ro)TNG#C9_{+*o9>^D2VI7Kv)kMVD`Q`+ITQ)GUHxndKqaV5>24vu?Ho|3bm=z-fM$>&Q?kL%x_7DU>@q z3vBhX_omYN;zGagrw7*n9W5OH)G;=lA`LiX zquu7}%;D}~Bce-=CCy|v)9UCac;{*06ETLS6wAHFRvyJgyirEJ^Q22|92LhCNIbZ$ zl+jveQCj%^R?FTEwY%^wFMX5YPWsXr$YoT)_74516%ze)>;&L^-TiMXqwvoBI~tzV zPMEROCVGt5>yk!lMcN!pS8JOEO<2SveM^OUkqmh^OP+0{`abe%Yk5|Cus%kznVQn@ zMl`1~GT64!=jPHO`xry~S`+m0j&|>R=*(ffHbWsKXE<70>z|t?2r}`%ze?H!Y4|H!~Tq^ zt>|pu;*|sf!SZ~yhwks*|4=6 zu*JleI_eVJZGl@kz?2*Ny~BVM=)4+~yk1iR8jm|&Ip#RV8taLEcO_Z7ReSiJo)uIz z9_@b*s0N=Ktf;FxFArR|-J5rc=K3gr=PRVBZdfK=QPBB>dP8_!4?})xC?c-vc(5FL zRMfB`ByjLX&8JNf#Z~9#JSGPAvQca7q{kcT)5iLU6!v*iTE}%;(B@@SRUYg~d&pes zN$8#6<2y_2PVICYki;oqlQ$|s_ z4AQDKBpeDy7JcyThBysediQ=Yto-^Ow($~+My`A})jEyAXpD?t&2IC~N8nT4VyjNQ z>MJKE>IPb~DxLkgnFAhIJtotNBaAYwmI~Cin=V~7QZtA*NmclP?QMyKE3Dp-!FD4z z#j4omzWc&^gR~hQz1nY1a_6Isj#rMd<_(G!t-rdWGNt0d%jI3W!{7BgJt}9_<2}@h ztoOw&2Hr&%WrVt|jHgB*Us)Z8F6WKJtImy{dj({dh;`~&ytB~Be#{XnAFnlyNY5I6t{-Ny{ED66pr-HD_SyQUchX*yqkB}#IGi*9 z{QdAgjQaiNc&2!0SD0XS4DFs~&l*U03y7{lXwRqw)jfR@ah21nTXLm@%z>7FbGF{4 ziIz4~Q_;zhO|?KS*Qj(Y8&>yqb*tG~HjnVCsZL*#5Yu|PVa;h}cOIpw18+XW(*btU z(e_F94bIz;5eH-{Yg)$*hZY#b^!EfEq+Y!feTBo96xS^3l3tRIs~;i= z52aalrBBrCFl>D1m@`z2EXZpUoQPp%q+h&a;lyjh{r>Qg3ed-cn%pacmAm>+6j=Ql z;0NCvmu{8xq&dS<9oYk&Gur<-zXmd|>vBE1b$Zn4;lBmZ?z}z~EWLI;V>W6kNzr@7 z#P!nO2<}{U(KvG>#eGW8uDkqt(V1sIB;p-gyP0ocBX7|m0Dp760V+kXnS?DHTNCQx zmS-{jFgR@z^*IxE%@_x)SrJCWQgrdCVKgB5Yr#p_R3>4P3Qi=gKze*2mD2j^ZRVSq zUu9kit!6TqDl=6&X-FCAnF@G3isF#5LiO(JGT+$x^kg{Ye&zU+&htUC<2KK}N{+$^ zU^~Lfy$)4^n1wse&4@VJCyV9WP5mrARtP4Kci(W{oe?GwLVDL${l!vi)vpb^9(+G_ zQNV*tSS~nc7{hytzjq)|sdibGF?IgWj9=wzIe5=jRSZ6lN(IqKLGl-un?mX-~AZ)=f4%N8BS+#c+oWidJ_CmrD-EW}dp- z<8j!rwk>q$a;)}#yxkL=o)jJ`CeKu^`J{`9HbVXieQAH6(PExu6%k~$u5~7J#I}m- z;HK$uUTkxU3n~J2G82`B|Hh?pWbGEmHJe>vhoXZH@EtaS-2)v%_zi~_IjfY4M<&wa ztAihAS|TB@rAh3MLxw>yHPFAP{2tQxdE^PF zP$f7R&!l|l8V$M-ep@4LpxU{SOH(kJ-OuZ(Eu+|Z9~{}Hxa_^9YEK)T(NE^tJ9?BW zq?$-a>sQ^gfD)7-SCB$6aRbeJH2~^(K?-9Zv|L!M-l;w{i<09{&1!9(&$ObtKDXa~ z-sIWIrsmeUo~v6B7szZ3E!RF@?P9#)H^x+LHW!poml0#+c8UxQahxk#vuc-obTV~6 zy6p|Rx&4^nqokU3(z%rHX58xg3%6CAkP%hP!^_M2WI3O@&8d{P5U!>`oGD&UunY|J zXw3|YgST-?j4+Dn4jv2-YAd#;l1N=hjC>QkNDK(wtB?UULs)S~+7o^iF~-9k6eKp! z&kPEDN(=R-g#+~g)6{ui)4;+A(a}1N;{%- zg;Pc!jg9C0ZQYA4+ch8H8;f9rl<7TY_SusIX&3;hC;$P#2Xf?i1VhG*N{nRIp>{;)RH(u6X<`D4pl;YgCaVR=BKv)ec z-5yaKoyoHKP-Jv0^R(axTiO1%*>)mVX!XgHnsW1k&8DjChr8H-yk_liK>jH}lRnqR3mT0olZEhaI{YKPyr z-PTZn(FdA{6FW3j(GX8(_CRGWqX#*^K@sjD2X%RgUA%oZ8+7#W?b6DWlwIATl#L@j zBOVFLQO6Q)f>9_vgsk_9>WW{ff%@GMvhTp-6dIEN4XosQ8Kb+JByeHMQ~ZJ;HG<(m zUA-b#Bn_$iGBbSC)TJ73Qf3|1M_6s6RHagkQsR6<2APXdE!CWdoJ z(GUQ-;Xu_heP%Fq^^43H1og;MyGO9*{iukLwU6x-SK8W0qrz+V!@JWyikvvpZ%TBg zJfxLJ29>3^=E#Vlh{tN|xhurj$(vOpizP8@vCfhPBrvV33Wp!Rs`RXS$*PTY3k+FW zH`dYE)*XAs5kez|43u28MeLgVj$bnnN@Bmi7R6<<5ADc%$P7$!zS5{Nd!0`kA7>a7 zb`O92`-EE`l4zQxM5j}uy1>0|bIosD%9R^YHaCEn&*&%^GM`?c%V{q%nqSi0SVZkl z8E&|hOJcJb0$$CyjkrCgs_JtJ6r2+GOw_->yfMrZ{XS1H>a;x(vT^&cjV=0YlW(q` z*(;!I>`9PnM0wx-N9nV>txvR9tQJ{D8#E84QcLqrouE2+07 z@rmH*A7Oi}-7BB|4cyj=Up=DE6${;V%5k(#TK&KmFTQY`udhRhFf|hBd0wzEI?_i- z)HYLUzntzgPv;kB>73q*4NAnflaM^L$uhcSGx(TQcMCEr-;*WwBLBHwwqi)JEm`EC zD=$6kh^v^|VXI#Zqbc1SvR?!-Jha%i46GYHkbI9_@5T#)w#IJ}FtM^w*q)J6m2HdO zpRQi2NPUR@)~K(~b(xBuw5it+UK%_{5jE?xgT(a|kF4%AI5BOPI}>@Hv4%7(4VfpQ zn?Pbd2yX({Ydb&*8p385i4?(m1R2SD+XNBGB{7qB_sfs$wnnp!vp!Vaxow-#)!;qJao|9KIJz^ICW=!$?>$ zV}1H;t?_a$M-(@6y5(*7^SplH`78N!wuaSOM9HD+b^%pIXWEr}-47WCINghffH@nM zG{$Pj<(#kjWH4h@IZn0Z+A73{`^Kr1yhvv2=*AmP2Tb_J$C8h#>-Z`|7gEf2j9CwR z@|9($)8Bv$A<_0JPPT42Y|(lmTNu@1vM-H=swrR~p+t#)9ze6ITo8y;%&AYv&A{4= z>1QW3RAfG3E_Jg}AeXAd$~rp_=`zciW;K$qw5U>&aIuQ}h0dRKWebV>Y;i#MLA3BmpRLVESmE}6l#=4jc(&%Ld1zTQ*mDSbfnssZB`#)j+G ziK8!s%I+;;Y}a=|lm4AkN^xhJ3cr{9HhLmyS^@MBgl4yOmJ?pg`=IbFDl=X_JMS0( zxutB6qv%KUg`KohTzEbrFVL;hjq2JSAYzQ zoBpbahrK5FZ3n_zE~!#4@Frc5ctaFO6dK`sJ7H5wN;o0W=9~I?IAPoul0o2s&>=>m zwW7?~cNbMtRjzGpkRI%}?Xks%`e!G?$mh=PEqbODtnH5BQ+cY+|&d+!1&N>ypndvDT02^|5Yx6lHF7Fr0s zLy~XBy^q^-o@ac&-}fEI9}Gv5wbop7&U?vtou8)s!>YZpr&^tG@tnR>^yg2$IAS{mkJMG_81*K3 zUK-!U^KM_Xf!2xvYs+?z@8WbsImgm>B&f7x#lH*J(@L+WqpNL}>RD;4EnT6=vfR zsQH-joWh0qmaBa)0MCos!u_`V*OBBo7unA17*O?3+wE7&*4fEVsdXUSL6{L59x>LCE`JKWur+#>@g12k3Ko-35nv(Eo!rS&eUwQy< zyaDXKPJEAo9}$!)(V!0%-9bZ+Zz;(R zoH86xw`hJ$zr>lVI#J(xofkx z7O^zOZ$*5eEplw8Wwe~Cy1gODUcc!5t|J1vu$&K-W?{ng={Uy>3opew)v{0%WHWv$ zifQVjL3`>jB4LU@u$0%7b@ozj{NI-H zcce@^G_IK7R-NvI-1*Ljxm@fwKQ;Nf>*Vw3!>;|xOaOeastk}0*4_7VI|Gke(wKp? zK|V3vM$doBA+=D?kh$8m6U3zKhesH`N4?5H?2}nY>B!`jQtxgWQuqW5nCD!r;-nG> z6Y-p&jQdQ5TbeVJ#OS8zeXjFWB`!f@dh(q_)3r)tJJRPZiRxUhL1a58^X}IIGhWg4 zR;jUOjl4?@PhO(-$X@u^8OxCjr9#~el>rvk)xbeo%BrxAXD(4)Pdro*#<6<-Iam7% z**;Q#BX*Cu_H2jo8Jb|FB&yWX@DXZB{>Yutr>A%9;K!u^s~eZAcgAWn`fG_%Rm+BlQ&BJNnn0!$3x=yp+$}r=7|y zk-{wPwlRx#@}LL!uyy-q0MyPyPmF5%qe$7I*_8|0@8gEOt3(GR)sCcv>Wk_}#xT*? zF+$}XVouZDVaZp~pGCY;$)uw{A4V}%tUTx>naM`o`dZ|(Aq3Msg^9!uK*}$ZzFjS8 z11#hP@d~bwYXP@b)l)C@JZDY3L*%(wkO%5{o^)8h_W}*R4!Y^_?a>^r%4rNIKV2Is z5b=mDK=>n_!-4G5cnJ;0G(l;c&eNvOKd7Z7)6|yNFr%t*E!Q*$&09nq093^HI2)&! zs8+>S*Qh%ljmV=I1pC%5ewW!SsTaPUuZh~_?B(*sdc4~5PBTRy#}KR4<>mVW?XRAL z4zH*SXstbPiA%2D)xPWH>JPOVEwxJC-zl`uGC+{9))BLA-R3Ag<;OeFMxrh6{g!8L z+zYCFZ|#~1V&Gv{@COA!d_HoveK&XGeS6nIFf_r@QL#h`?FzilC=nSk9hdyrq*lsO=(7i z*bU5mMZdjLzwWnqjDXkG7{7IwXJzuCud$?g1Wt||H(qUx_MBn_wd%OC#F)$&m@xOv zzod4V?W)AvTm_Ae_ql_qx$|hgTKN@pl0#A130m?T$K&L78@9{%G<^J(6`h>hadWd( z9E2Hx6n1@Ar?ReELAxg{iOlWl_Wf`s;!N8o^078)Y!l}3ed-e~_o|fh)$Z#TTd6a^ zWZ9iFzXaNs#OXz>ak8>By`qdd@pX*STU=AlHx1_H$Mxlv*b%e4>#Gi9-(^sSb!8*% zHD-FG(Bh4wGX zxY>lTDtk2c$-=staBt>X6vJIQC1A2m{Hn{VCyHw4`XD)7i(hFd#wR>Gb6R>Y5QVQh z>Pa~&wC#_2qsJD!;_owOdv5UViLfGfbKFLFzv2^Re=XD!uA`*&elkv&r;eeeLQzEV zd!~g=K6c&Hb{>;YmUe3Ug`BI#In6eiMNR}yOH%Mw zSNpms%<2&5ke2rJLziXYMAOW`lb?ac8&WY%8y=|fB>AC}s|SVF$=-v}kChFXS3l;j z>b;sUZb}KW;!MccQJ2rmn~jJtr$Ri@m7nr)<&@Pbd9~{jkQH0EmXqrQ9H77NPau%9 zhjxt~tZ{~DDi%!b5hqbU*2@M@p zQ-2so_i7u})vjN}qRlr7chuR6nKSdWut3tNEKzZE*eQj3|FZKTBBjTo>GcV9lxu*_ z%>Y(B?hFY&SMUpMuF^)QGjL(BCx{=zD9sN^MR91-QWWl=FQq9=`p zZs|Cc+ZccF){;qu&+%2z!z!zx;TJFhVY@VtE}OrgOHhQemq~Jjy8tf>4pN*bebI3q zX0<=WyH&c;HMfvn9G9X-wcL?Ia)vszC>b+5BPoR@uian*#mEDhTnX_8RvR_W03QbL zwu1%+OyGjYh@a-Hk{NJ3QS>Dq+20Xk(5)kCxM4FSNeU!0ZZKau?;V)jU(Se1Nhfm# zKoSi{T@ZswqefJY7ek?b(z7*jo;W+M4amH~RxRo&;G4GJ9B^a?;}EtV(Ph1j)ujhV zO3Ed|Zp(Yj z|8@ZA7XoQf8#oGmy#J{B8TkNNh5(Hx9m$it+;`Io8|;NnPRxX}Q}IOu04G17xvI=%KkKyw7%D1x-+mz*rE7iKNWO zf1<}Dj{;aJRGR&}_H@P2Ja%Niy22HRyt5mFnZYeYQ|HO5o}P|mK<#R4X5p_d;2)Ar zY~a1nu(T@6c?wJ7Hy2Ye6#lGin)erfbUB&IjkT?XzC&#-Hvehks(kp|>4YrcpqvZ% z0x+*>mP6kMQpk%c8w{nUTy9kl4ax3v1JIRvgnL<8)sziQO0M4f&ux-Ddwu~Se;9}= z-*>%$aqC?Yw9~J=q&#lBgFiU(>S1%*(nMaFlH}EukN2mFOMe*3X9KsnNCW_3s>GA| z_h~s|tsGLnSpr%7YXO#=jSwYCufuh#nFLRqz;6660)vcm>{)V-r7DbnfgxF6%#Llo zK4w?Zq|r;)`k|Cf1@zY~0;@Sh^_c2X1B1L`keBP7)5=E&SKs|YrV?h&TL75Uzwn^n zZ>tb%{53FGA_zR^;~?FyzLY*+w;r=#l-cvYz((%^I4{~eG&kVkknwkfbcHl9&9?PmO{|jvN9v&C_-xvIi+x-`q@Bg$BM8!Eyu;r|vVDJi) zHF3#*HS?dj6W{`nb;~hnH21?HHoDDIKdFFHi{7j|3H7MU&GCOfBc<{M_-#SU^t0nW zRh}AGUOa7+Mq_iBHsT48Bro1!lOw@fJk^caBfSeNzu0N>?gJByqLM5BDt>}BTLF-8 z`&H$hq>6L502>v>Bg9LZ^&P)fz+nz#v4(>8Y-j8U0Pi=e8+%Z=2zBO!~w#0D8rH_aBXkKTLcPIOWFr-|POzTJnEhzf9n&-Ijs5nS!Xk_HANM0gu&wHx$Xk*5Jru~j$$JY-F2@1Ey-n{-0$mYlp7)X7eHRBCI z2&Kwr3k|LCg}!2NOFqA$hP!q{Lp{dLzkj#h#Z@yItu6^$RnIN>@EgMa&#HW1CFC59 zTK39KV4D69tGr(Ud>8-ozt{c8G0Xq<%YWy~A13t6tp009|6M2k(+&IoUnfM{++=Jk z2c^>C%U~bX5IV>I>4*b77FW4#f;CPL&=Lit+C@KPI@bQ!=YP+^?92FH0qkiK)udV_ zkJ}?_@GeT|?Qw@7lGYvnePkSa9WD8vlpcEIM<&yTy2W}`1?}@Z*6~yji2{I@m0sv! zR)Q<;p)=tPi`M}WsEBuT#W~K6@q9!R{~W^~NpIdE1zomA(o$;#YuY%)02hboA0*WG z-?(|6a7`vneT2TMF2YJu^N$@Rk0gMid(=&)3gA2By{aLRZ83g5|2mDp+r*a>1)4mS z{LFaQOvNxj33w=5EN7uz$KpR+SZ_oxTrSeVzSQXs=bNcHZ-}wKYRyWJs0mIT34^AGAainkNx*Qwq}dWAMY|yxf;Q z{KXy7JKtW1ALi*RHP`cFP)f0`+G4D~rNV$OxULlG)jsI)1h8h7Ik!gaAowoJRi?A1 z6#8*koGp*uMA1Ief+l|af)C_l$Se>L+po`Q)MB_8&HiaZ2f0o|iG)?Z@S%;u8-gmJ z(^%ry%Q|W4dv{Cg)!uxN#J9nh*yj+g|CYkXdX9SEP}S;>V}qhGMA(Bq_{)f}PkMBX z>4I4R!$y+ z1^;$`Tg|{%0?vjf>_r}9C5F^wX)?vsbrNz9v-GO%Yu;2Y&fuCi^FAT3$wTQd|nDGYZkyAHQp+SoULkQ`Fw(-y}#8; z$@rfAWMX%hS98L@2VnrDMWWXoJ@`-|U|A;SoU8`$>A&VSK|pPV`TOhjhMc&Kfh`&} zefx$gl6%hwW;_oKq28@;&(t2{&zP^uXdp`YUFsQLvyD5Wc|*7^7?vU!3cS+%LGB;)j7YMt zxV54uWtc-mlh4OdT{#ySBI-Rr@(5C52%U_k1Xr$Bj2)PaPPpX=;AgFBKmg`s!o7NG z$S4(8AdLNiksLF}v+exr#ct?;MQvAchbL!Sc13;3f~8Fa&f@0ye_rgkx+OFg>DRYO14NdF3 zJtJ_DDXy5Xt)6b3bCE}QM+PS@BfvU==LlRmzPXngLHsvo4gl8b$+wp_@^>fIg#UQ2 zF19q@y)m%wUE#vI`us6%D;F70_%KPX_ey?lL#+aytt6b`$x8F)I|{$G?QOBn5$$kI zPpi=1cJxM+jA+D7i(0Q``W@_h7S#hP8nv0%qYia;E!O?HJzrn;_r@=TBS=yofOTNt4uN#3cg?Z0dTOOErwHOsOwJ!xahY3H)b@2%GSm&-)z+g< zHAGJlH0jVrf;I=8>OMbwFSuDwWZ!sKKL{@&C5j+ zKEP@~h%@+;gV~lKd7138j3N4%Zw-660&|q4UA?UH~^gFSBEa!Re~5}3whTLwa4rw^M7gW z{5?AqZ`|33;=Z}Z*592m%Z3(TUL{)j#A@t1X_-4QC{1S@8uz&D=0oY*I#Iv24$Rdv zIH~e=w!^qG|MIcrzI;T8+CMZ$fCqG0h=2R0_iTz_k-UR@%trZxS7d1t1s}!$>QXCx zo3eM|zDGCltIPxI3V=%PMjj}j?GVwA-OtOkfpt+Ffcc7jbxbj^FI0c*1D_O|g@2s2 z&TgkHz#8qGy_V}Xf+qs~do>G~ zC+H5MOflTwT!w>0jA;J2Zx9@e-JyhA6&9~oy(30{mAOX z6x&Y2Cqfc7TKGhkUgI0PY3N$LbFpdg^@AVajmkpKhBTyh2-Tl=6`$2^>n}G?O?f3v z>zM^en6gS<(_(qF6%M|bK5+GMH^n%hSGEpCU`m9~w%Q;TKX08bRw$#J8xFb*KicEX ziH6vRZzXASyrfw8q$hgopn<_ujYdg-$p? z?cOWJ#8QiOk>Lq&IGlE`Rt8~40JYE8oIRfH5V#i`JYPPJG3y06-ptId?yZgXis07d zz3oPA5_wGPFuPjG#rlWI`mfEH|K!giBDBLTY@#D^o@O;2T*@Gt>uN0D1VYuWR_c+x zjYgI0Da?+WMpglIW}Hdh-DXXdrli8$kc;4719AJ#L8yQADU;$=HvSXr&r#F3X%w_| z}Gty^|G-IoMiK?$)~93E^Un$p+6Kf8wLH8VEwN|EY03Rl#R4=ei1o&C3h zZmB;Kz^{kpgxC4^H4=~4roIjI*Vm2c*_X?97o5xfBkOoyY&O0VXSJN_XHxT`KMYU@ z)M?f0u_`!6%41(JRp-$!v>TS}xirbQfmwS`jc!GpLB5lJ#Ov%-J`U)Vj;P%9LFvCt zCmladwQb)a)z>0o{*&JGX+ZEMHMOf$GL^xbY%bP|8t+TPKCxcN_)>$6)n5btB? zebnIR1~2t9qqeJg8L!547vJ>cIDxXZ*|oo&kS>w^0Q7;$y+-;*A|i4%Ce zIbKP`aV5d4?ZJ2jK^v0BScx|`)sQOJhMr^ghS7kaAn*X($T06#QH5leI{ zT7hSaWh3Dn)6ifdOv`7C%^F!G%rBjV{trV= z4^o--(X%ltoQ)?1B*HiX@AZjQvz~w~z#s!AZ+q-O^wBjqUk{xhyM8`N)Bel;Ft-2} znm=xhBJ2}3%Tqi-1Nl$HVS&+00m% zP8`j~d;$}3K3j*v?ZU5!0|e1&bA%BqWBt679uK`=cZBjHKYko7&}4&W_WO}6i@y6O z;?#TJtPS?HHBWJP+wcUv`mI^p&^5-;?-~aCxqg&md%FV6e`nN|VUxkt}{#AcO}KW^uD(%gHwLBbL0owfT_spe8u>5{;Jr zyDVZ$#WF}>XSLutF@GIJVHwE&aZqbxUrR!XlKQg?DEAI z;KSwxltsKvw@sj%Wu5C73O1X5+G$&0w`7av`ib`ss_4I=A_ONR3?UUvDc&=$FEJhQ zEj3udUEzZQE_$a@jg!m_+5fn978(I8QB30>LUE<+UvUNKJC%l=5j4~@+ar0juo6D| ziNe4mfCka#`-hu@m|VDhMR>Av>-IsL|8X4aq7q&uO?3CFg6ArOAm^Qf`V}-oi)QY1 z-o;C8>z%me+=FnE#=&}`hwi_pV!(THopO)Q01v%2dd|`P{XPg#ufyw9R@eq2*50lG zo4Ka2H579I$2&p9_DFSz^t~ft=&UrT+{h5BRYv{_&p@dGHr^h}c0=arqi3#=wf4AA z<;NvEbWOM&sY(=r?>PPAepu)PfTkUir@c>^PiHZQ3hunCb4YCiN69uXZu9olPX*;z z^Hks0aYkPG(*WHs#0tiWx-pgM1t@&M%Ytc58r)}UyRzv8@k3fi;C_iltqE!x)&OyM z`PC8E7Eu}1|GdBMaK4=sHht@#P@Oxrd2tr)X=>LnW8`_<{%t;()+-M)PucVhbFhFE z?5$!Npob~!coC=CEqHIWRFy_+JU!X0h@^qx?hC%0ZGO#4F@Yb=7)qV}?M}CLI_b$D z-kT&`@Onjk!IPu+B>j8m&N2UxeIcgdUqy#o6hD29Qe*|6bNXXs?*zOfUV{5iA5kZ6 z4fwuHuAXv|7u>AhwE;XD5>u5zifoC3`0AdAkwPZ!Q;xyHX9E%kKmBp;IteAog7D@G z2i57pG_M8iptf9bpVslqZ`WlZ5#KsPzLWj~@hvH}cWCz&y9-#P4z}aDzZr3Zh zfBw|(oA2)jMv^Tt!+{)h(y(Ep!FviP>cX@w^N zWdifi+w526tG1=6C(>KCTIeqwR$n-n8lJRm^bC)VHNl$~ zwv*Y>&F7^jKX)cs2om-uJos%`z_zSqJQ)=U2X8hFrz9PH@ZD~wd|653Hr~Qfza=gR z=)?>$ws~pZ034~SKXl8FHjSf4@@6|H@Pak!xk}*~kO>@^&=7WkM3k*qgMM1Bj9rn5h#aj}pup*8!b zN}Ttt`GPxcf>FeywBcw9cNQ^Y7H4R)HQV9_2$aC98~2N^#=TjrxC%J6nnfX~NIH;+ z4_B%{PV7bdyqzE^W%N{W;r{AG_X!#%xRN#r2lwDX1p^O=M6iow!$JLYUOOy_JM)v5 zlnNer?@jR4CvWD>>Z%j|w!rg*kG5J6h>A-=5qYu;o7L~^>u+ZO`ehw}M1wIPl9@zu z_ZeXx)oq|t-t$F(D_PWY#w(2%n~6E=jX9gn${UWC60!3Rb*8}wqi(~!MT!pA;8Wff z#FA~11ROHjXsC}GDM@y{mIHT!BfRVY5JA8sxrgnxiT43gm)Chl$Q##Ib%=348NCfL z9HS|=K1>;HShj<7&YbGzvNswgQlx48Gl|S7JiNCvX4ySC$LlA!(EFawHrqJwX{}54 zrP>ae>3G!&)MS;l?k+=F*JWVsxH}e-dvby*g{vzIQ_PI!@*S)S;^Rk)7Xf{S5K`I? zjZyoPnUUxjZ!rAjvLV&y0D<2rJy)i!q1|DdK6$ZABSrC$*(z%CSJxwfd5W^GDdjqN|=pa&&4_J|FQsgW4fQH)~B7}l%Ju^~-HR*;t z-fXCD9|0|DTNnpnw@FVQ)ifw~Rs!u(i+mUnXp0|WC-w>0%SMf+an=f$PnYoVYyR_Dy$Yf#Mn>B zhE7P^vlJjI0kn|efFzTt@5$cB($>Sv;4Z@_YjcVCL%iKRC%_iO?7 ze3Y=WJ?2gie%JrYVYawJFb#Dl6-+tVN0|dVQxza(UisEUy(1b3kh0&J@Zf(J5KEsn zs802?02F=;_CwXbGEXX;1Kc+;QvOQ|F39Yk_UqH^fKa0Vt|;IB_Xoe< z;Mep6<}zPDU-kmRs(hN`gs=T*R7@ZSgizuu+e zX}qD5_*^&@0U480B_LNY4hYMWzpe&C6Mdi-VGAUKD}l_pR_5hJJeY}Fb{0s>f0GQo zJ<$`-WsB^JfuJGqNjJDHCDs)^StVjl#}I@|5n zPZjYk_)Osmcw;2Qj{){)K-iZRiYkHH{QSb;DPM+J%W8Un@6^BxABaA)Jqw_gf<-uM73QKVGy42n%lcJ zTJ{(DU)omyAkh9+Cq-K{ST>;Y^TBQ}T3?V5*Prclt0v~FwQIn8-kC=5 zGoIiw4ybU-N!L<;ZKyH$J$?V1Ou) z431z??HpHzzIT8YL5-{k8*`d=#wmUG<@2rv$DkEJ42%ZEl7ru|Lx_+1t5C2)?VPi{ z92L-tmN5ZA_~SGsUEFOfy~O*s_byz#_Cd#^#`6Vm#1bC^y(6kID{XCll*%>>@)@P8 zS{SX{Y9*e|GW6Yt&3)VE>KEdw$A}ZV5KlA*T*%b+wBG9n#PV)-tWMQB7wDhE2;l{( z_ls~zR+w6} z7qV*TK3!#uNos;$acvm!?u?z8?-aZ|>;gqj(kT`J&r(gXMOD zlhBaoVnJa^IbN8Oo!Z9pCIX-K+n2XHVDyB%pelBYWNEJk;En>+Jna{7Zh^vc0Lbfb=~pb65%A%_U!D4}5-mglJb&Zw9`9)9A64!QYI>Xp^l(FBWO!<9**xYSdlm0<*j?mUHD zlo@`=+pZMR19`vu9v#gIvmlwhxMzBORxNOoj3cpyN9OU3--k)5sQGuNb^s@jJ`5En zYngk6gT(hD>1!}&2v~}J3_M^V|2SA)zB@?j%*(Rf+y@Y8whp{#*-{z(dP~B%H;{;U zelT&*5b2H_Sq4gt@GO_17iyU3##Hc7CB#~U69Hl=Sz@rQ=0=**03NO<0rsg_=cE%2 zb@E$(+QDrG#jQw1+0KU`#Tc+zpSy^&CFGrL?{Txr*gHBShgYbP z)O1AD%B5-uLw=0v(=N83s%9{mrxUqXp2c{V@dkyx(nP0WTIk9sTzVGC#U8dASaflX z0S-EuoxZlet+TB+`hwjl8wTiHwjhztaOybI0vsPzZ#EiPomeRxe4S=ZT2Z*zG=v!N zTwJ1vK+=)4H0*~9(p>Ss+z%P1fl9x}COjE${M%W_q0h(+fE|Aj_NYA)oURs_u!#+P zFZG;;TVRyA1na`n-(m0`OnVt!Vb{oJU3aU1 z+n}x>agB_0n=LdX%7uc5wshPbCzk@W&^1O&n<7Zcs_an+MDQb9i<<1Hn-;|?RCu#W96@l||0!k;#H?xx&Yo4uetqR>zR!8O6Z>!1| zB=%YQdC4x{EM6|nbbdKhOE2=}#Y5FEfe}zYFM+W~u!pu@T@Mw7qf=5#6O{zONAHmW zO@&F%9FNzyST?hEy&Ii1)e^-{t*6ux7`Hb&5u2o}sp9Hujl8et{HqW(b%l?&Zy@@= z*vvbgmDg47!W2&-O%YvD=BKvM2QFFl38#pw2In~O=q!YHno(13b9+J1uhx?LLS2>J|=L!=?Ia$8>EjV8z6ha>pDV~z?^aSP87 zyjoNxHYbmOj$63;Ju(zq)}jc#egGtRxXLORQ&aqXl)Q@^J=eHaJt&nc#vMU3@nj&- zN$0rF@?GtwqxMt}J-z;Z<<*~j;DJb6+*$I-$6FvHCPN3u)lUDj?kB54%ha)`Gg?mb z^oUFGM@q*6hx4}Y#m!)lE%q2cH4C}vsG;rUd=bByW5{RlvdL+}&|&u)7EHdD_Hi@S z5{G+)?kw^1i`VST!c4@lBd|-=(48uI13z6XsP4JXul@fK@n1I2tti@-K#WL z09+>;5t}KJIW0S}3yiP}%+_6})>t z1P#uT;%c}8+f?>HQRiN6Jl$aP(3h=z0I5q#9U-1eTN+k0K}xTFbtl4R0BXVBoWJ#XPz2A_7lMk*lL3ar&7;g_Z%NQ6Mi zZn)1bQFitY)BzczUH5Zk7}QBbLonz0NbgRfidPFMc6@5h2gcv~E{!a6Rsw1^Z%>&F z+E3dwnR9PCHJs6Ofj80$Z3cS3{BUaYvmfyL1edj1eX6UE?Q=8|85NT;PS$WM>Sf|d zP*7j0MOoA43ocW#N!-yoDAQg~RXxx$#%N4@V>mDBQlk)6HEQk?dHR#i%U^%M&0u-6W<}B9b8+S(&qv?$Op<~IyL>X%+9_d{l|XfXPGt(@Af|oiDNukWy(h9#@m0F5~8DtGOS~* zpgX&HS1)(q2QA2GsL~&?SII?pH)BE{*3-pc3OV;k~WAf?<4@(+SqPE9kx1+TK|`rf?yjjQLweS9vkkr|fXV?YD~y%$nkQ?`afZxznI1-~{^w|G9hKaBvcwt;DEpTqMPU0W3)&P?~4T9W_&!GUFw@V(6%#-#o-;;=o}Ei zlLXyY;>giFeoDZB7vqcHU5zvzJF#3#j5-JY74kX}^8PsPpd{>Ko)rL(tW3F%8KQxj zmJE!U)AXjCQh{P3FGKE}doz*tP%Gb5wyewQQ1+mem$PDaAS4RAcR^W-ha9FObzr$~dC2qw_y=j9Ob>j*5DYe^vki!5Z^mqRM`58eLx`2wSt-`RYtM z-;P~m@smUW_QEq;rgyl9hOY5XO{GRGGrTTl3Mf=NK+{aiBj6jinWwFM_XeedT}Mui z17}W_10xQ-P*w?P3sB-Z-=TP@r!L49^^zwN!&*VS_FPtf6NnO^7E5qRU@2+Z?Mj?p zkJ{E3g}SWX5^vuI^LCjPRj__rg4~$n1SM}rHJWrRF49B5 zvav9;kA`nAUuo_L$50h+z&u*EJi0)J&llWp0-~2=TOG9*_wQI}zX~6^om<Vf7BOjXUsU=Fyha{u<4{CzeTSMEXz)Iaaf2q6qw#3o!Nr2--H@BwB6a#&SAl7{_)w&s>-viFLoh{P zHR&WvZqgU$qU)4|xT9oIjuidagW)Ebp^?whmV#?TvH9a(SedDultLe<$uVqb$8n(p zN!|$xUVi*zS8{t8$Q4fKY^zFc`!Xv(eb%c5Z8f~3NUhpQ($t&T$JFaAhNi0~UBMML z^%S?^wWi*P@}9Sw=w51{y!`662T*(F>ij5eQF5J{2Qp8}9nNR7V1^XLdu+r6CxeAI z3$W;ZDvWMnI#6Da49!tuna;_B7h$~G9~PL(RkNIsZby`=ce&#p$R;>3A!8t!94vO~ zOSKVibH3aDbac7yP8pM+N7_C{RA3^;DZ%1y0pQ|lqaJ@v@rRmh-OEpM#a#L})-5@l zbajUa7!iw$KwdE6eDdQ&@a5pULBtW{lf1&ec&A60u9IAR$&;pw;aGt1g>#- z*=KbuRhFlO<*MAv)V1f8r+S&K%Dx?X)sHa*9&HK%4e_Pe>w$4@ZcPe`$sDnz{-`*q zKJM%p_H6?muf3HI8t|_0K&9QglGhmRnlNrS>v@-Y-%mAaPq?rO1>(sGSkv=6*}2Et zs#?0z(zTTsKF>9XPE!$b2K#4) zr!0?O7V8&v>VKi;%X1=qDl-Xkb*W2x?zbG4h=}g@Shh~bOgF*GRg941aWRbt=C}t_ zKmmvBH7b%@9m)X?c1(X({wup{?28Aj@x4@{AwD9(T4@gZ@5;wYeM`}N;yC~POUcsV z#J@7M_0x)Q2uuvs)^f>3ieVe#+T+k$)E-DtK<2kIzN3l*m5NTPw>@K%jvwjJ8 zXM+xoas86B2^2Y>ptF=d9=XYd(51$;5BlFW6{0z87(e!n6h3e$e4qSUcK?Ic7UNWH z4MR$y9grL-WqgfYNlha1Y`o_>qRKD7cMT#g2Iua@sru{Js_GvPg>N<;kM0VVoMAwS zlXpuf_E?UZo9_=1{!NKy+to%=#9w~}0Y4rC)qvN}9*h)-CUl^qltpy+=TH0Z4p2-T z!Pc?}^2Kub?yO!-nn}vKG{VcUvR3QAuh*|EyRf*NEd`}peskga7G)yh$5W2gJVFS$s%gOFW8s0P@02tA;;yw8hbT&siSg`hd&~cD-`=&;_ zpNhHf^%$xA0`*K4l@#pm@fp0ET)El^rISCXg0)`W&DLE(gRxHyIuA_D)M@pWj}{_H zyVLs0k9Oal?S`dPsP{af1%ZX9&bX6rwfAjq^Fb+429BO6euE7ISf(^7A5_w31^{uX z^7E{9B9sOL3COI1zY9`_+QPYQ&-v3dg-2_U+n$=7=1sC-aoysl{%W+_o?dK~#9iRk z(`uCF+Jp^1J<)%&x%z^O2ft5lBwJ9cV~3ngK{Xw=u({V z0^a@V2Wn7(W?Cs5UrTqscdxonqI$vYYnzdf9ea@4vCFkXMNNpwTzNM~sFv#S{KFS6 z5l#^W(ofsOyA7p~B26gzL$8QTVk=T|)QU=CD@Z%pxsRX23nnrgc3qw|Y*NT?0;39& zMJ*>TZ+cHc0H^Yu$*K9-S?ZFm0u~z=UEO5(<)A<7`Mah3;Ym6(S3s^&8V738t_JxH6V??nsGciqe^p>SA~}v!Ws;v0R%npCx8?f|wsg2)cM{S8HmnEqJ*oG3T1B=gDs1^~@x0 z%^BlGSVG#%8pfLN0WeR}b*feCz?%xF_A4{%l{0k->FE33%W11mO*3Sd8 zRTiWAzDbeoG}}xdIcq7vqKHJ7GMzp`JzjRtUbUA{L?uO&SJKQMGZima-~F<7Gm^|b zgmdih=s>o>T}{Z}=&U}~rQ~p|pRd@XF$j`MH@`nvr^m$7baV_hQ@AQ?d6DJ?3cV|P zYToD;0uJVevow+zoMqusWnGPbY0roXoNq9Di|>4o3-p>cGHUa7Ems6B;3I~cjed*)=7EWZ5dh+k}V zHED53?U=it(>s0lO~k}@U4f9Cv?XhkhZhIhhlX%lz}nZ{=;JAiJqVYD&ah3cA}L4Da6>u^pi*LJT7H951BW59j{vGAB~oCG90;T-1Q0d@HNnl zX@Y}bjX%rnm*myq?P%p~bg2(92Iq(zGEYE1;zX!y?+Jqwv&q5p_@FiX@h2Zfn^!m2 zQnc^Si2i_@Tp>QYD3Kr)S1xBoc)D6+Irij`T1(dRsGFF_&VVQ8eYZsFyFjYe?TZm~ zW2tUP_GqI~vy;IXUVrV|#i^Sirb7O@->G*vl86+{3ppI8Z zQ?@nhkwuPpS@vuCC5pl8 zbhZw|eLUI10kCE8vYNmDpwD5^Q^jTCP)g40sO-IcO8+J8MvvnqwnlmR)I#mjzG|z= zQYH*&s6MCCz7oM^G=%evkv@$cWg$P?$(>M=lefJK%JQ6vX%Yb_gMg{pPNbLLRB&Wf z6d>AtQ{_@u>pS8$0#va;6?RJ&b4TSuMQzubcWrS|M79%AT;u7SW{JzMg4QR|vz8I$ z^VqtD*rC$z0B5Q%5k`iCVP$TK>vhkEFU*}~@Z>13!wthq3AV19=C|^kjfQ$vRut`O zodY~$24uedd~^139$v@uS;?D2CEpK}^xwzU7&(Myu9`7&9D&%9@?*^=_%Pi2{x6c# zIchL0)(ISbvC1_1dk^jqxxQZ&r}_o?Ppc=Oj^^W6lSn%z8ac0tN>*u9%LmDvRVF+$ zv@Dg&MMMjb0R^}<>HFQJ)0I^vdya($oxb7sS=}++n!M(QeS566hOF_B3X95qxv#O# z{;Uw;Gd`cHO=wnDR+fS<@+;-EjaO6+|Ue_Kd(YgWn7Wfwi`6BcF z6y8@qK9x7MJQFt|z<1Ssz$0SKF#h#_e7F62m%si1CG)9rZ4$GcKR)~YlMjR|pF)`b z@ Date: Fri, 2 Dec 2022 19:09:36 -0500 Subject: [PATCH 02/19] Add design details Co-authored-by: Katrina Verey --- .../3659-kubectl-apply-prune/README.md | 297 ++++++++++++++++-- 1 file changed, 273 insertions(+), 24 deletions(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index 563a8e1045d..ce7b5f88e5f 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -58,7 +58,7 @@ If none of those approvers are still appropriate, then changes to that list should be approved by the remaining approvers and/or the owning SIG (or SIG Architecture for cross-cutting KEPs). --> -# KEP-3659: kubectl apply --prune redesign and graduation strategy +# KEP-3659: ApplySet: kubectl apply --prune redesign and graduation strategy -When creating objects with `kubectl apply`, it is frequently desired to make changes to the config that remove objects and then re-apply and have those objects deleted. Since Kubernetes v1.5, an alpha-stage `--prune` flag exists to support this workflow: it deletes objects previously applied that no longer exist in the source config. However, the current implementation has fundamental design flaws that limit its performance and lead to surprising behaviours. This KEP proposes a safer and more performant implementation for this feature, along with a plan that will enable it to progress out of alpha while continuing to satisfy the needs of the users who have come to depend on it over the past 20+ releases. - +When creating objects with `kubectl apply`, it is frequently desired to make changes to the config that remove objects and then re-apply and have those objects deleted. Since Kubernetes v1.5, an alpha-stage `--prune` flag exists to support this workflow: it deletes objects previously applied that no longer exist in the source config. However, the current implementation has fundamental design flaws that limit its performance and lead to surprising behaviours. This KEP proposes a safer and more performant implementation for this feature as a second, independent alpha. The new implementation is based on a low-level concept called "ApplySet" that other, higher-level ecosystem tooling can build on top of to enhance their own higher-level object groupings with improved interoperability. ## Motivation @@ -176,6 +175,7 @@ When creating objects with `kubectl apply`, it is frequently desired to make cha - MUST natively support custom resources - MUST provide a way to accurately preview which objects will be deleted - MUST support namespaced and non-namespaced resources; SHOULD support them within the same operation +- SHOULD use a low-level "plumbing" object grouping mechanism over which more sophisticated abstractions can be built by "porceline" tooling. +A v2-prunable "apply set" is associated with an object on the cluster. We define a set of standardized labels and annotations that identify the “parent object” of the apply set and the “member objects” of that parent. We operate at the plumbing layer; we aim to support: + +- listing the parent objects efficiently (porcelain may expose this as listing groups of objects as managed by the various tools) +- listing the member objects for a specific parent object efficiently (porcelain may use this for advanced diffing and pruning, and for presenting objects grouped by their higher-level aggregation) +- basic apply-with-prune operations, where it creates or reuses a Secret in the cluster as the parent object. + +### Apply Set + +"Apply set" refers to a group of resources that are applied to the cluster by a tool. An apply set has a “parent” object of the tool’s preference. This “parent” object can be implemented using a ConfigMap, Secret, or a CRD of the tool’s choice. + +“ApplySet” is used to refer to the parent object in this design document, though the actual concrete resource on the cluster will typically be of a different Kind. We might think of ApplySet as a “duck-type” based on the `applyset.k8s.io/id` label proposed here. + +### Member Object Labels + +Objects that are part of an ApplySet should carry a standardized label, with a key of: + +```yaml +applyset.k8s.io/part-of: +``` + +The `` can be chosen essentially arbitrarily (subject to the limits of label values). + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> Should we identify a few patterns to avoid collisions? Either uid:12345 or :... +> +> <<[/UNRESOLVED]>> + +### ApplySet Object Labels + +ApplySets should also have a “parent object” in the cluster. This ApplySet object can (in theory) be of any type. For performance reasons we later propose limiting to ConfigMap, Secret and custom resources with a specific label. In future we may define a common CRD, but we believe we can achieve a reasonable user experience without defining one. Many existing tools avoid using a CRD, so that they can be used by people without the cluster-admin permissions needed to install a CRD. (This also avoids CRD versioning problems etc). + +The ApplySet object should be labeled with: + +```yaml +applyset.k8s.io/id: +``` + +Implicit in this are a few assumptions: + +- An object can be part of at most one ApplySet. This is a limitation, but seems to be a good one in that objects that are part of multiple ApplySets are complicated both conceptually for users and in terms of implementation behaviour. +- An ApplySet object can be part of another ApplySet (sub-ApplySets). + +How the ApplySet object is specified is a tooling decision. Gitops based tooling may choose to make the ApplySet object explicit in the source git repo. Other tooling may choose to leverage their existing concepts, for example mapping to a Secret or ConfigMap that they are creating already. The tooling is responsible for consistently specifying the ApplySet object across apply invocations, so that pruning can be done consistently. + + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> Do we agree: For kubectl specifically, we propose supporting but not requiring explicitly provided parent objects, with automatic object creation in the latter case. This is explained in more detail below. +> +> <<[/UNRESOLVED]>> + + ### User Stories (Optional) @@ -355,12 +438,175 @@ Consider including folks who also work outside the SIG or subproject. ## Design Details - +### Efficient Listing of ApplySets + +In order to support listing all the applysets in theory we would need to query all GKs with a label selector. However, we can reduce the set of GKs that need to be queried with two optimizations: + +For built-in types, we limit to ConfigMaps and Secrets. +For custom resources, we require that CRDs that define types that can be used as ApplySet objects be labeled with a label with a name of `applyset.k8s.io/role/applyset`. + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> The value is currently unspecified (TODO: maybe v1?) +> +> <<[/UNRESOLVED]>> + +A `kubectl apply list-applysets -n ns` command would therefore do the following queries: + +```bash +kubectl get secret -n ns -l applyset.k8s.io/id # --only-partial-object-metadata +kubectl get configmap -n ns -l applyset.k8s.io/id # --only-partial-object-metadata + +for crd in $(kubectl get crd -l applyset.k8s.io/role/applyset); do +kubectl get $crd -n ns -l applyset.k8s.io/id # --only-partial-object-metadata +done +``` + +Optimizations are possible here. For example we can likely cache the list of CRDs. However, while the number of management tools may grow, the number of management ecosystems is relatively small, and we would expect a given cluster to use only a fraction of the management ecosystems. So the number of queries here is likely to be small. Moreover these queries can be executed in parallel, we can now rely on priority-and-fairness to throttle these appropriately without needing to self-throttle client-side. + +In future, we may define additional “index” mechanisms here to further optimize this (controllers or webhooks that watch these labels and populate an annotation on the namespace, or support in kube-apiserver for cross-object querying). However the belief is that this is likely not needed at the current time. + +### Efficient Listing of ApplySet Contents + +We want to support efficient listing of the objects that belong to a particular applyset. In theory, this again requires the all-GK listing (with a label filter). An advantage of this approach is that this remains an option: as we implement optimizations we may also periodically run a “garbage collector” to verify that our optimizations have not leaked objects, perhaps `kubectl apply fsck` or a plugin. + +We already know the label selector for a given applyset, by convention: we take the id from the value of the `applyset.k8s.io/id` label, and that becomes the required value of the `applyset.k8s.io/part-of` label. + +In order to narrow the list of GKs, we require the applyset object to define the list of GKs in use. The plumbing tooling can optimize selection of the objects in this applyset based on this list. + +“Porcelain” tooling can still perform tooling-specific GK identification. Tooling generally can use their existing mechanisms, be they more efficient or more powerful or just easier to continue to support. However, by using the standardized labels proposed here, they can interoperate with other tooling and enjoy protection against their resources being changed by another tool (such as kubectl). Tooling is not required to implement these labels, and we are not introducing new behaviours for “unenlightened” objects. + +To identify the GKs in use, kubectl and applyset-compatible tooling shall add an annotation `applyset.k8s.io/contains-group-kinds`; we use an annotation instead of a label because the annotation can be larger, and because we do not need to select on this value. The value of the annotation shall be a comma separated list of the group-kinds, in the fully-qualified name format, i.e. `.`. An example annotation value might therefore look like: `certificates.cert-manager.io,configmaps,deployments.apps,secrets,services`. Note that we do not include the version; formatting is a different concern from “applyset membership”. + +To avoid spurious updates and conflicts, the list must be sorted alphabetically. The list may include GKs where there are no resources actually labeled with the applyset-id, but to avoid churn this should be avoided and ideally only be a transitional step during an apply or prune operation. + +We may in future define additional mechanisms, such as supporting a field selector on the CRD that identifies a strongly-typed list. + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> Any we want to do now? +> +> <<[/UNRESOLVED]>> + +Where no list of GKs can be determined the tooling should warn that we are performing a full-GK scan. As discussed in the interoperability section, tooling should not populate the annotation unless it believes itself to be the manager of an applyset. + +In pseudo-code, to discover the existing members of an applyset: + +```bash +for-each gk in $(split group-kind-annotation); do +kubectl get $gk -n ns -l applyset.k8s.io/id # --only-partial-object-metadata +done +``` + +### Cluster-scoped ApplySets + +We need to support ApplySets that are cluster-scoped, for example ApplySets that include installation of CRDs (such as cert-manager). The mechanisms we have defined here work for cluster-scoped ApplySets. Today’s tooling will create an managing object in a namespace, and likely a cluster-scoped CRD would be more intuitive than a namespace-scoped resource. However, no additional explicit support for cluster-scoped ApplySets is required or proposed at the current time (but cf the next section for cross-namespace considerations). + +### Cross-Namespace ApplySet Contents + +When querying for ApplySet contents, an ApplySet could contain cluster-scoped resources or could contain resources in other namespaces. Querying for this content is generally going to require more permissions and be slower, so we would like to avoid over-querying here. + +Best practice is likely to avoid ApplySets spanning namespaces. However, sometimes this is unavoidable - particularly when managing cluster-scoped objects - and the “plumbing” tooling cannot enforce this restriction. + +Where a GK is known to be part of the ApplySet and is cluster-scoped, we should naturally query for those objects at cluster scope; any permission problems here should be surfaced as errors. Where we cannot determine the list of GKs for an ApplySet, we may support “discovery”, likely warning that the ApplySet does not define a list of GKs, and then attempt to perform cluster-scoped queries, likely warning if there are insufficient permissions. + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> Should kubectl ever fall back to this? Will it have a separate subcommand for it? +> +> <<[/UNRESOLVED]>> + +For GKs that are namespace scoped, we would normally expect those to be part of an ApplySet object in the same namespace. We define an additional annotation however for cross-namespace ApplySets: `applyset.k8s.io/additional-namespaces`. The value of this annotation is a comma-separated list of the names of the namespaces (other than the ApplySet namespace) in which objects are found, for example `default,kube-system,ns1,ns2`. Note that there is no need to include this specifically for cluster-scoped objects, as those are covered by the group-kind list. We reserve the empty value. If this annotation is present, the tooling will query namespace-scoped resources in those namespaces in addition to the namespace of the ApplySet object (if any). If this annotation is not present on a namespace-scoped ApplySet parent object, the tooling will query namespace-scoped resources only in the same namespace as the ApplySet parent object. If the annotation is not present on a cluster-scoped ApplySet parent object, the tooling will not query namespace-scoped resources at all (kubectl will output an error if given namespace-scoped GKs in this case). + +As with `applyset.k8s.io/contains-group-kinds`, this list of namespaces must be sorted alphabetically, and should be minimal (ideally other than during apply and prune operations). + +As cross-namespace ApplySets are not particularly encouraged, we do not currently optimize this further. In particular, we do not specify the GKs per namespace. We can add more annotations in future should the need arise. + +### Kubectl Reference Implementation + +We will develop a reference implementation of this specification in kubectl, with the intention of providing a supportable replacement for the current alpha `kubectl apply --prune` semantics. Our intention is not to change the behavior of the existing `--prune` functionality, but rather to produce an alternative that users will happily and safely move to. We can likely trigger the V2-semantics when the user specifies an applyset flag, so that this is intuitive and does not break existing prune users. + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> Add summary of exact commands and flags proposed +> +> <<[/UNRESOLVED]>> + +The proposal may evolve at the coding/PR stage, but the current plan is as follows. + +We will add a flag to kubectl apply, `--applyset=`. If specified, that will change the behavior of apply and prune to include the new functionality. + +We may also develop additional `kubectl apply` subcommands, such as the `fsck` functionality (perhaps `applyset-verify-integrity`), to build complementary functionality that is helpful for adoption. + +We intend to treat the flag and any subcommands as alpha commands initially. During alpha, users will need to set an environment variable (e.g. KUBECTL_APPLYSET_ALPHA) to make the flag available. + +When `--applyset` is specified, kubectl will automatically create a secret named `applyset-`, in the targeted namespace. kubectl will populate the labels/annotations on applied objects as described here. + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> Will we not support cluster-scoped applysets initially? There is no +> obvious choice of cluster-scoped built-in resource to use for them. +> +> <<[/UNRESOLVED]>> + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> Will we also support an object being provided as part of the input +> resources, and if so, will we limit the kinds? +> +> <<[/UNRESOLVED]>> + +When pruning with `--applyset`, kubectl will delete objects that are labeled as part of the applyset of objects, but are not in the list of objects being applied. We expect to reuse the existing prune logic and behavior here, except that we will select objects differently (although as existing prune is also based on label selection, we may be able to reuse the bulk of the label-selection logic also). Dry-run will be supported, as will `kubectl diff --applyset=id`. + +We will not support all the combinations of flags that apply and prune currently does with the new `--applyset` flag; this is not a breaking change and our goal is to support the existing safe workflows, not the full permutations of all flags. We can add support for additional flag combinations based on user feedback, in many cases the meaning is ambiguous and we will need to collaborate with kubectl users to understand their true intent and determine how best to support it. In particular, we will not support the `--selector` flag in combination with `--applyset` until we understand the intent of users here. Flags associated with the prune alpha (`--prune`, `--prune-allowlist`, and `--all`) will also specifically be excluded. + +We will detect “overlapping” applysets where objects already have a different applyset label, and initially treat this an error (we may add “adopt” or “force” functionality later). + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> Expand on how we will accomplish this: +> - Do an extra GET request before SSA? +> - Use a separate SSA manager to create the labels on the objects to get automatic conflict detection? +> - Is there another option? +> This has consistency and performance implications. +> +> <<[/UNRESOLVED]>> + +We will not support “adoption” of existing applysets initially, other than by applying “over the top”. Based on user feedback, we may require a flag to adopt existing objects. Note that adoption is not trivial, in that different users may expect different behaviors with regard to the GKs selected or the treatment of objects having/lacking the `last-application-configuration` annotation. + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> Expand on adoption (taking over management of a set created by another tool) +> vs migration (taking over management of a set created by the old pruning mechanism) +> and how kubectl will or won't facilitate these flows. +> +> <<[/UNRESOLVED]>> + +### Tooling Interoperability + +There is a rich ecosystem of existing tooling that we hope will adopt these labels and annotations. So that different tooling can interoperate smoothly, we define some requirements for safe interoperability here. + +For read operations, we expect that using different tooling shall generally be safe. As these labels do not collide with existing tooling, we would expect that objects installed with existing tooling would be invisible to the porcelain tooling until they had been updated to include the labels. We do not propose to implement “bridges” to existing tooling, rather as the proposal here is lightweight and small, it makes more sense to update the existing tooling. We may add warnings such as “applysets using an old version of X detected, upgrade to v123 of X to work with those applysets”. + +For write operations, we need to be more careful. Deleting an applyset using the “wrong tool” should be safe, but we will likely include a confirmation if deleting an applyset using the “wrong tool”, particularly unknown tools. We expect that porcelain tools may define richer behavior on delete, so this is the equivalent of pulling the power cable on an applyset instead of performing a clean shutdown. + +We do not believe that update operations are safe if using the “wrong tool”, because that tooling may have additional metadata that would then not be updated. Tooling should generally reject applying on top of unknown applysets. PorcelainTooling may choose to recognize other tooling and implement specific logic there; in particular this may be useful for moving between different major versions of the same tooling. We may implement a `--force` flag, but this would likely be logically equivalent in outcome to a full applyset deletion and recreation, though with the potential (but not the guarantee) to be less disruptive. + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> How should we recognize that we’re using the “wrong’ tool? +> Should we have something like `applyset.k8s.io/tooling: helm/v2.0.6` on the applyset? +> +> <<[/UNRESOLVED]>> + + +### Security Considerations + +Generally RBAC gives us the permissions we need to operate safely here. No special permissions are granted - for example there is no “backdoor” to read objects simply because they are part of an applyset. In order to mark an object as part of an applyset, we need permission to write to that object. If we have permission to update an applyset object, we can “leak” objects from the optimized search, but we can support a “fsck” scan that does not optimize the search, and generally the ability to mutate the applyset carries this risk. Using a more privileged object, such as a secret or a dedicated CRD can limit this risk. + +Known Risks: +- A user without delete permission but with update permission could mark an object as part of an applyset, and then an administrator could inadvertently delete the object as part of their next apply/prune. This is also true of the current pruning implementation (by setting the last-applied-configuration annotation to any value). Mitigation: We will support the dry-run functionality for pruning. Webhooks or future enhancements to RBAC/CEL may allow for granular permission on labels. + ### Test Plan @@ -881,11 +1127,14 @@ Why should this KEP _not_ be implemented? ## Alternatives - +### OwnerRefs +We could use ownerRefs to track applyset membership. A significant advantage of ownerRefs is that pruning is done automatically by the kube-apiserver, which runs a garbage collection algorithm to automatically delete resources that are no longer referenced. +However today the apiserver does not support an efficient way to query by ownerRef (unlike labels, where we can specify a label selector to the kube-apiserver). This means we can’t efficiently list the objects in an applyset, nor can we efficiently support a dry-run / preview (without listing all the objects). Moreover, there is no support for cross-namespace ownerRefs, nor for a namespace-scoped object owning a cluster-scoped object. These are not blockers per-se, in that as a community we control the full-stack. However, the scoping issues are more fundamental and have meant that existing tooling such as helm has not used ownerRefs, so this would likely be a barrier to adoption by existing tooling. We do not preclude tooling from using ownerRefs; we are simply proposing standardizing the labels to provide interoperability with existing tooling and the existing kube-apiserver. + +### ManagedFields + +We could use managedFields to track ownership, however again this is not standardized and the kube-apiserver does not support an efficient way to query by managedFields manager (today). This too may be an interesting area for porcelain tooling to explore, and we should likely be defining some conventions around field manager names, but that is complementary to and out of scope of the current proposal. It does not appear viable today to define an approach using managedFields that can be implemented efficiently and in a way that is adoptable by the existing ecosystem. + ## Infrastructure Needed (Optional) From 2fbdc7c1d6a10519d0f2820e42f52c150c9c485d Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Thu, 22 Dec 2022 14:01:16 -0500 Subject: [PATCH 03/19] Add another open question --- keps/sig-cli/3659-kubectl-apply-prune/README.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index ce7b5f88e5f..39fc9cedc21 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -185,7 +185,6 @@ know that this has succeeded? - MUST NOT formalize the grouping of objects under management as an "application" or other high-level construct - MUST NOT require users to manually/independently construct the grouping, which would be a significant reduction in UX compared to the current alpha -- MUST NOT require server-side API changes - MUST NOT require any particular CRDs to be installed - MAY still have limited performance when used to manage thousands of resources of hundreds of types in a single operation (MUST NOT be expected to overcome performance limitations of issuing many individual deletion requests, for example) @@ -522,6 +521,21 @@ As with `applyset.k8s.io/contains-group-kinds`, this list of namespaces must be As cross-namespace ApplySets are not particularly encouraged, we do not currently optimize this further. In particular, we do not specify the GKs per namespace. We can add more annotations in future should the need arise. +### Objects with owner references + +> <<[UNRESOLVED @justinsb @KnVerey]>> +> +> If an object in the set we retrieve for pruning +> has owner references, what should we do? This would be +> somewhat unexpected; it implies that kubectl apply doesn't +> really own the object, apparently. +> We could consider this an error, or perhaps warn +> and skip the object, assuming GC should take care +> of cleaning it up. +> +> <<[/UNRESOLVED]>> + + ### Kubectl Reference Implementation We will develop a reference implementation of this specification in kubectl, with the intention of providing a supportable replacement for the current alpha `kubectl apply --prune` semantics. Our intention is not to change the behavior of the existing `--prune` functionality, but rather to produce an alternative that users will happily and safely move to. We can likely trigger the V2-semantics when the user specifies an applyset flag, so that this is intuitive and does not break existing prune users. From dee73a2afe267edea29ed0ef11be89e328d291b0 Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Tue, 10 Jan 2023 19:24:50 -0500 Subject: [PATCH 04/19] Links, clarifications, ownerRef and GKNN explanations --- .../3659-kubectl-apply-prune/README.md | 60 +++++++++++++------ 1 file changed, 41 insertions(+), 19 deletions(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index 39fc9cedc21..613e6788cd4 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -278,13 +278,22 @@ For a more detailed walkthrough of the implementation along with examples, pleas - **GVK allowlist mismatch**: the allowlist is hardcoded (either by kubectl or by the user) and as such it is not tied in any way to the list of kinds we actually need to manage to prune effectively. For example, the default allowlist will never prune PDBs, regardless of whether current or previous operations created them. - **namespace mismatch**: the namespace list is constructed dynamically from the _current_ set of objects, which causes object leakage when the current operation touches fewer namespaces than the previous one did. For example, if the initial operation touched namespaces A and B, and the second touched only B, nothing in namespace A will be pruned. - TODO: link issues +Related issues: +- https://github.com/kubernetes/kubernetes/issues/106284 +- https://github.com/kubernetes/kubernetes/issues/57916 +- https://github.com/kubernetes/kubernetes/issues/40635 +- https://github.com/kubernetes/kubernetes/issues/66430 +- https://github.com/kubernetes/kubectl/issues/555 #### UX: flag changes affect correctness If the user changes the `--prune-allowlist` or `--selector` flags used with the apply command, this may radically change the scoping of the pruning operation, causing it over- or under-select resources. For example, if they add a new label to all their resources and adjust the `--selector` accordingly, this will have the side-effect of leaking ALL resources that should have been deleted during the operation (nothing will be pruned). On the contrary, if `--prune-allowlist` is expanded to include additional types or `--selector` is made more general, any objects that have been manually applied by other actors in the system may automatically get scoped in. -TODO: link issues +There was also a previous bad interaction with the `--force` flag, which was worked around by disabling that flag combination at the flag parsing stage. + +Related issues: +- https://github.com/kubernetes/kubernetes/issues/89322 +- https://github.com/kubernetes/kubectl/issues/1239 #### Scalability @@ -292,21 +301,24 @@ To discover the set of resources to be pruned, kubectl makes a LIST query to eve A related issue is that the identifier of ownership for pruning is the last-applied annotation, which is not something that can be queried on. This means we cannot avoid retrieving irrelevant resources in the LIST requests we make. -TODO: link issues - #### UX: easy to trigger inadvertent over-selection The default allowlist, in addition to being incomplete, is unintuitive. Notably, it includes the cluster-scoped Namespace and PersistentVolume resources, and will prune these resources even if the `--namespace` flag is used. Given that Namespace deletion cascades to all the contents of the namespaces, this is particularly catastropic. Because every `apply` operation uses the same identity for the purposes of pruning (i.e. has the same last-applied annotation), it is easy to make a small change to the scoping of the command that will inadvertantly cover resources managed by other operations, with potentially disasterous effects. -TODO: link issues +Related issues: +- https://github.com/kubernetes/kubectl/issues/1272 +- https://github.com/kubernetes/kubernetes/issues/110905 +- https://github.com/kubernetes/kubernetes/issues/108161 +- https://github.com/kubernetes/kubernetes/issues/74414 #### UX: difficult to use with custom resources Because the default allowlist is hard-coded in the kubectl codebase, it inherently does not include any custom resources. Users who want to prune custom resources necessarily need to specify their own allowlist and keep it up to date. -TODO: link issues +Related issues: +- https://github.com/kubernetes/kubectl/issues/1310 #### Sustainability: incompatibility with server-side apply @@ -314,6 +326,9 @@ While it is not disabled, pruning does not work correctly with server-side apply One solution to this would be to use the presence of the current field manager as the indicator of eligibility for pruning. However, field managers cannot be queried on any more than annotations can, so are not a great for an identifier we want to select on. It can also be considered problematic that the default state for server-side applied objects includes at least two field managers, which are then all taken to be object owners for the purposes of pruning, regardless of their intent to use this power. In other words, we end up introducing the possibilty of multiple owners without the possiblity of conflict detection. +Related issues: +- https://github.com/kubernetes/kubernetes/issues/110893 + ### Related solutions in the ecosystem The following popular tools have mechanisms for managing sets of objects, which are described briefly below. An ideal solution for kubectl's pruning feature would allow tools like these to "rebase" these mechanisms over the new "plumbing" layer. This possibilty could increase ecosystem coherence and interoperability, as well as provide a cleaner bridge from the baseline capabilities offered in kubectl to these more advanced tools. @@ -507,7 +522,7 @@ When querying for ApplySet contents, an ApplySet could contain cluster-scoped re Best practice is likely to avoid ApplySets spanning namespaces. However, sometimes this is unavoidable - particularly when managing cluster-scoped objects - and the “plumbing” tooling cannot enforce this restriction. -Where a GK is known to be part of the ApplySet and is cluster-scoped, we should naturally query for those objects at cluster scope; any permission problems here should be surfaced as errors. Where we cannot determine the list of GKs for an ApplySet, we may support “discovery”, likely warning that the ApplySet does not define a list of GKs, and then attempt to perform cluster-scoped queries, likely warning if there are insufficient permissions. +Where a GK is known to be part of the ApplySet and is cluster-scoped, we should naturally query for those objects at cluster scope; any permission problems here should be surfaced as errors. When a tool cannot determine the list of GKs for an ApplySet, it may support “discovery”, likely warning that the ApplySet does not define a list of GKs, and then attempt to perform cluster-scoped queries, likely warning if there are insufficient permissions. > <<[UNRESOLVED @justinsb @KnVerey]>> > @@ -515,7 +530,7 @@ Where a GK is known to be part of the ApplySet and is cluster-scoped, we should > > <<[/UNRESOLVED]>> -For GKs that are namespace scoped, we would normally expect those to be part of an ApplySet object in the same namespace. We define an additional annotation however for cross-namespace ApplySets: `applyset.k8s.io/additional-namespaces`. The value of this annotation is a comma-separated list of the names of the namespaces (other than the ApplySet namespace) in which objects are found, for example `default,kube-system,ns1,ns2`. Note that there is no need to include this specifically for cluster-scoped objects, as those are covered by the group-kind list. We reserve the empty value. If this annotation is present, the tooling will query namespace-scoped resources in those namespaces in addition to the namespace of the ApplySet object (if any). If this annotation is not present on a namespace-scoped ApplySet parent object, the tooling will query namespace-scoped resources only in the same namespace as the ApplySet parent object. If the annotation is not present on a cluster-scoped ApplySet parent object, the tooling will not query namespace-scoped resources at all (kubectl will output an error if given namespace-scoped GKs in this case). +For GKs that are namespace-scoped, we would normally expect those to be part of an ApplySet object in the same namespace. We define an additional annotation however for cross-namespace ApplySets: `applyset.k8s.io/additional-namespaces`. The value of this annotation is a comma-separated list of the names of the namespaces (other than the ApplySet namespace) in which objects are found, for example `default,kube-system,ns1,ns2`. Note that there is no need to include this specifically for cluster-scoped objects, as those are covered by the group-kind list. We reserve the empty value. If this annotation is present, the tooling will query namespace-scoped resources in those namespaces in addition to the namespace of the ApplySet object (if any). If this annotation is not present on a namespace-scoped ApplySet parent object, the tooling will query namespace-scoped resources only in the same namespace as the ApplySet parent object. If the annotation is not present on a cluster-scoped ApplySet parent object, the tooling will not query namespace-scoped resources at all and should output an error if given namespace-scoped GKs. As with `applyset.k8s.io/contains-group-kinds`, this list of namespaces must be sorted alphabetically, and should be minimal (ideally other than during apply and prune operations). @@ -523,17 +538,15 @@ As cross-namespace ApplySets are not particularly encouraged, we do not currentl ### Objects with owner references -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> If an object in the set we retrieve for pruning -> has owner references, what should we do? This would be -> somewhat unexpected; it implies that kubectl apply doesn't -> really own the object, apparently. -> We could consider this an error, or perhaps warn -> and skip the object, assuming GC should take care -> of cleaning it up. -> -> <<[/UNRESOLVED]>> +If an object in the set retrieved for pruning has owner references, +tooling should verify that those references match the ApplySet parent. +If they do, the tool should proceed as normal. If they do not, the +tooling should consider this an ownership conflict and throw an error. + +We are taking this stance initially to be conservative and ensure that +use cases related to objects bearing owner references are surfaced. +In the future, we could downgrade this stance to recommending a warning, +or to considering owner references orthogonal and ignoring them entirely. ### Kubectl Reference Implementation @@ -1141,7 +1154,16 @@ Why should this KEP _not_ be implemented? ## Alternatives +### Full GKNN listing + +Instead of encoding a list of GKs to scope in, we could encode a the full list of GKNN object references, making the ApplySet parent object a (somewhat) human-readable inventory of the set. The reason for not choosing this approach is that we do not think it would actually allow us to further optimize the implementation in practice, and that its additional detail would make it more prone to desynchronization. + +The reason it does not optimize performance in practice is that we're considering the source of truth for membership to be the `part-of` annotations on the resources themselves. This is useful for visibility and for ownership conflict avoidance, but it means we must retrieve the objects themselves to check the source of truth rather than relying on the GVKNN. Since individual GET calls are far more expensive than LISTs in the common case for pruning, in practice, we would end up extracting the GK list from any GKNN list and make the same calls we would have with just a GK list. If it is deemed worthwhile, we could indeed do this, and it would allow an additional layer of in-band drift detection via comparison of the precise list to the set of current labelled resources. + +Alternatively, we could omit the `part-of` label entirely (which leaves no means of ownership conflict management), or consider the GKNN list the source of truth (which leaves a much wider vector for object leakage in practice than GK listing does, in our opinion). + ### OwnerRefs + We could use ownerRefs to track applyset membership. A significant advantage of ownerRefs is that pruning is done automatically by the kube-apiserver, which runs a garbage collection algorithm to automatically delete resources that are no longer referenced. However today the apiserver does not support an efficient way to query by ownerRef (unlike labels, where we can specify a label selector to the kube-apiserver). This means we can’t efficiently list the objects in an applyset, nor can we efficiently support a dry-run / preview (without listing all the objects). Moreover, there is no support for cross-namespace ownerRefs, nor for a namespace-scoped object owning a cluster-scoped object. These are not blockers per-se, in that as a community we control the full-stack. However, the scoping issues are more fundamental and have meant that existing tooling such as helm has not used ownerRefs, so this would likely be a barrier to adoption by existing tooling. We do not preclude tooling from using ownerRefs; we are simply proposing standardizing the labels to provide interoperability with existing tooling and the existing kube-apiserver. From d1a6bae1393f9f3e9c740a47c55d2cb019d13e8c Mon Sep 17 00:00:00 2001 From: justinsb Date: Wed, 18 Jan 2023 10:48:21 -0500 Subject: [PATCH 05/19] Follow-on to initial feedback, address some unresolved blocks --- .../3659-kubectl-apply-prune/README.md | 173 +++++++++++------- 1 file changed, 105 insertions(+), 68 deletions(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index 613e6788cd4..3c65493fa45 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -297,7 +297,7 @@ Related issues: #### Scalability -To discover the set of resources to be pruned, kubectl makes a LIST query to every GVR on the allowlist, for every namespace (if applicable): `GVR(namespaced)*Ns + GVR(global)`. For example, with the default list and one target namespace, this is 14 requests; with the default list and two namespaces, it jumps to 26. An obvious fix for some of the correctness issues described would be to get the full list of GVRs from discovery and query ALL of them, ensuring all previous objects are discovered. Indeed some tools do this, and pass the resulting list to kubectl's allowlist. But this strategy is clearly not performant, and many of the additional queries are wasted, as the GVRs in question are extremely unlikely to have resources managed via kubectl. +To discover the set of resources to be pruned, kubectl makes a LIST query for each Group-Version-Resource (GVR) on the allowlist, for every namespace (if applicable): `GVR(namespaced)*Ns + GVR(global)`. For example, with the default list and one target namespace, this is 14 requests; with the default list and two namespaces, it jumps to 26. An obvious fix for some of the correctness issues described would be to get the full list of GVRs from discovery and query ALL of them, ensuring all previous objects are discovered. Indeed some tools do this, and pass the resulting list to kubectl's allowlist. But this strategy is clearly not performant, and many of the additional queries are wasted, as the GVRs in question are extremely unlikely to have resources managed via kubectl. A related issue is that the identifier of ownership for pruning is the last-applied annotation, which is not something that can be queried on. This means we cannot avoid retrieving irrelevant resources in the LIST requests we make. @@ -335,25 +335,25 @@ The following popular tools have mechanisms for managing sets of objects, which #### Helm -**Pattern**: list of GVKNN (from secret) + labels +**Pattern**: list of Group-Version-Kind-Namespace-Name (GVKNN) (from secret) + labels Each helm chart installation is represented by a Secret object in the cluster. The `type` field of the Secret is set to `helm.sh/release.v1`. Objects that are part of the helm chart installation get annotations `meta.helm.sh/release-name` and `meta.helm.sh/release-namespace`, but the link to the “parent” Secret is somewhat obscure. The list of GKs in use can be derived from the data encoded in the secret, but this data actually includes the full manifest. #### Carvel kapp -**Pattern**: list of GK (from configmap) + labels +**Pattern**: list of Group-Kinds (GK) (from configmap) + labels Each kapp installation is represented by a ConfigMap object in the cluster. The ConfigMap has a label `kapp.k14s.io/is-app: "”`. Objects that are part of the kapp have two labels: `kapp.k14s.io/app=` and `kapp.k14s.io/association=`. Getting from the parent ConfigMap to these values is again somewhat obscure. The `app` label is encoded in a JSON blob in the “spec” value of the ConfigMap. The `association` object includes an MD5 hash of the object identity, and varies across objects in a kapp. The list of GKs in use is encoded as JSON in the “spec” value of the ConfigMap. #### kpt -**Pattern**: list of GKNN (from ResourceGroup) +**Pattern**: list of Group-Kind-Namespace-Name (GKNN) (from ResourceGroup) Kpt uses a ResourceGroup CRD, and can register that CRD automatically. The ResourceGroup contains a full list of GKNNs for all managed objects. Kpt calls this full list of objects - including the names and namespaces - an “inventory”. Each object gets an annotation `config.k8s.io/owning-inventory`, where that annotation corresponds to a label on the ResourceGroup `cli-utils.sigs.k8s.io/inventory-id` #### Google ConfigSync -**Pattern**: list of GVKNN (from ResourceGroup) +**Pattern**: list of Group-Version-Kind-Namespace-Name (GVKNN) (from ResourceGroup) Distinct sets of synchronized resources are represented by RootSync / RepoSyncs, along with a ResourceGroup that has the full inventory. Each object has some annotations that define membership, including the same `config.k8s.io/owning-inventory` as is used by kpt. As with other solutions, following the “chain” from RootSync/RepoSync to managed objects is somewhat obscure. @@ -382,11 +382,17 @@ applyset.k8s.io/part-of: The `` can be chosen essentially arbitrarily (subject to the limits of label values). -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> Should we identify a few patterns to avoid collisions? Either uid:12345 or :... -> -> <<[/UNRESOLVED]>> +We recommend that tooling uses a prefix that is its "tool name", e.g. `kubectl.` or `helm.`, +followed by some tooling-specific unique value. We can investigate maintaining a formal registry, +but we initially reserve the following prefixes: + +* `kubectl.` for kubectl +* `helm.` for helm +* `kpt.` for kpt +* `uid.` for where the suffix is the UID of the applyset object (e.g. `uid.e049464e-4583-4642-9649-93dcb0e96bd4`) +* `id.` for where the suffix is the group-kind followed by the namespace followed by the name, such as + `id.configmaps.ns1.parent1`. (While this is a little tricky to parse, it should be unique because + neither namespaces nor object names allows dots.) ### ApplySet Object Labels @@ -406,11 +412,9 @@ Implicit in this are a few assumptions: How the ApplySet object is specified is a tooling decision. Gitops based tooling may choose to make the ApplySet object explicit in the source git repo. Other tooling may choose to leverage their existing concepts, for example mapping to a Secret or ConfigMap that they are creating already. The tooling is responsible for consistently specifying the ApplySet object across apply invocations, so that pruning can be done consistently. -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> Do we agree: For kubectl specifically, we propose supporting but not requiring explicitly provided parent objects, with automatic object creation in the latter case. This is explained in more detail below. -> -> <<[/UNRESOLVED]>> +For kubectl specifically, we propose supporting but not requiring explicitly +provided parent objects, with automatic object creation in the latter case. +This is explained in more detail below. @@ -456,14 +460,12 @@ Consider including folks who also work outside the SIG or subproject. In order to support listing all the applysets in theory we would need to query all GKs with a label selector. However, we can reduce the set of GKs that need to be queried with two optimizations: -For built-in types, we limit to ConfigMaps and Secrets. -For custom resources, we require that CRDs that define types that can be used as ApplySet objects be labeled with a label with a name of `applyset.k8s.io/role/applyset`. +* For built-in types, we limit to ConfigMaps and Secrets. -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> The value is currently unspecified (TODO: maybe v1?) -> -> <<[/UNRESOLVED]>> +* For custom resources, we require that CRDs that define types that can be used as ApplySet objects be labeled with a label with a name of `applyset.k8s.io/role/applyset`. + +The value is currently ignored, but implementors should set an empty value to +be forwards-compatible with future evolution of this convention. A `kubectl apply list-applysets -n ns` command would therefore do the following queries: @@ -486,7 +488,7 @@ We want to support efficient listing of the objects that belong to a particular We already know the label selector for a given applyset, by convention: we take the id from the value of the `applyset.k8s.io/id` label, and that becomes the required value of the `applyset.k8s.io/part-of` label. -In order to narrow the list of GKs, we require the applyset object to define the list of GKs in use. The plumbing tooling can optimize selection of the objects in this applyset based on this list. +In order to narrow the list of Group-Kinds (GKs), we require the applyset object to define the list of GKs in use. The plumbing tooling can optimize selection of the objects in this applyset based on this list. “Porcelain” tooling can still perform tooling-specific GK identification. Tooling generally can use their existing mechanisms, be they more efficient or more powerful or just easier to continue to support. However, by using the standardized labels proposed here, they can interoperate with other tooling and enjoy protection against their resources being changed by another tool (such as kubectl). Tooling is not required to implement these labels, and we are not introducing new behaviours for “unenlightened” objects. @@ -494,13 +496,9 @@ To identify the GKs in use, kubectl and applyset-compatible tooling shall add an To avoid spurious updates and conflicts, the list must be sorted alphabetically. The list may include GKs where there are no resources actually labeled with the applyset-id, but to avoid churn this should be avoided and ideally only be a transitional step during an apply or prune operation. -We may in future define additional mechanisms, such as supporting a field selector on the CRD that identifies a strongly-typed list. - -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> Any we want to do now? -> -> <<[/UNRESOLVED]>> +We may in future define additional mechanisms, such as supporting a field +selector on the CRD that identifies a strongly-typed list; we do not plan +to do this in the alpha. Where no list of GKs can be determined the tooling should warn that we are performing a full-GK scan. As discussed in the interoperability section, tooling should not populate the annotation unless it believes itself to be the manager of an applyset. @@ -524,11 +522,11 @@ Best practice is likely to avoid ApplySets spanning namespaces. However, someti Where a GK is known to be part of the ApplySet and is cluster-scoped, we should naturally query for those objects at cluster scope; any permission problems here should be surfaced as errors. When a tool cannot determine the list of GKs for an ApplySet, it may support “discovery”, likely warning that the ApplySet does not define a list of GKs, and then attempt to perform cluster-scoped queries, likely warning if there are insufficient permissions. -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> Should kubectl ever fall back to this? Will it have a separate subcommand for it? -> -> <<[/UNRESOLVED]>> +For the alpha scope, this functionality will be restricted to subcommands of apply +(like the migrate functionality). We will likely want a command similar in spirit +to `fsck`. We will add warnings/suggestions to the main "apply" flow when we detect +problems that might require a full-scan / discovery. We may extend this based on +user-feedback from the alpha. For GKs that are namespace-scoped, we would normally expect those to be part of an ApplySet object in the same namespace. We define an additional annotation however for cross-namespace ApplySets: `applyset.k8s.io/additional-namespaces`. The value of this annotation is a comma-separated list of the names of the namespaces (other than the ApplySet namespace) in which objects are found, for example `default,kube-system,ns1,ns2`. Note that there is no need to include this specifically for cluster-scoped objects, as those are covered by the group-kind list. We reserve the empty value. If this annotation is present, the tooling will query namespace-scoped resources in those namespaces in addition to the namespace of the ApplySet object (if any). If this annotation is not present on a namespace-scoped ApplySet parent object, the tooling will query namespace-scoped resources only in the same namespace as the ApplySet parent object. If the annotation is not present on a cluster-scoped ApplySet parent object, the tooling will not query namespace-scoped resources at all and should output an error if given namespace-scoped GKs. @@ -536,6 +534,15 @@ As with `applyset.k8s.io/contains-group-kinds`, this list of namespaces must be As cross-namespace ApplySets are not particularly encouraged, we do not currently optimize this further. In particular, we do not specify the GKs per namespace. We can add more annotations in future should the need arise. +Where an applyset includes both cluster-scoped and namespace-scoped resources, +by reducing to the above cases. The set of relevant resources is determined +by consulting the `applyset.k8s.io/contains-group-kinds` annotation; whether +those kinds are cluster-scoped or namespace-scoped are found using the normal +API discovery mechanisms. Cluster-scoped resources ignore the +`applyset.k8s.io/additional-namespaces` annotation, namespace-scoped resources +combine the current namespace from the applyset object (if that is itself +namespace-scoped) with the namespaces from the annotation. + ### Objects with owner references If an object in the set retrieved for pruning has owner references, @@ -548,6 +555,17 @@ use cases related to objects bearing owner references are surfaced. In the future, we could downgrade this stance to recommending a warning, or to considering owner references orthogonal and ignoring them entirely. +### Versioning + +The labels and annotations proposed here are not versioned. Putting a version +into the key would forever complicate label-selection (because we would have to +query over multiple labels). However, if we do need versioning, we can introduce +versions by including a prefix like `v2:` (and we would likely do +`v2:[...` or `v2:{...`). Colons are not valid in namespaces nor in group-kinds, +so there is no conflict with the existing (v1) usage described here. Labels cannot +include a `:` character, so if we needed to version a label we can use `v2.`, +however our usage of labels is primarily around matching opaque applyset-id +tokens and thus seems unlikely to need versioning. ### Kubectl Reference Implementation @@ -576,12 +594,10 @@ When `--applyset` is specified, kubectl will automatically create a secret named > > <<[/UNRESOLVED]>> -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> Will we also support an object being provided as part of the input -> resources, and if so, will we limit the kinds? -> -> <<[/UNRESOLVED]>> +In future, we may support a applyset object being provided as part of +the input resources, but we will do so in response to user demand and +user feedback, and do not have existing plans to do so in the alpha +scope. When pruning with `--applyset`, kubectl will delete objects that are labeled as part of the applyset of objects, but are not in the list of objects being applied. We expect to reuse the existing prune logic and behavior here, except that we will select objects differently (although as existing prune is also based on label selection, we may be able to reuse the bulk of the label-selection logic also). Dry-run will be supported, as will `kubectl diff --applyset=id`. @@ -589,25 +605,34 @@ We will not support all the combinations of flags that apply and prune currently We will detect “overlapping” applysets where objects already have a different applyset label, and initially treat this an error (we may add “adopt” or “force” functionality later). -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> Expand on how we will accomplish this: -> - Do an extra GET request before SSA? -> - Use a separate SSA manager to create the labels on the objects to get automatic conflict detection? -> - Is there another option? -> This has consistency and performance implications. -> -> <<[/UNRESOLVED]>> - -We will not support “adoption” of existing applysets initially, other than by applying “over the top”. Based on user feedback, we may require a flag to adopt existing objects. Note that adoption is not trivial, in that different users may expect different behaviors with regard to the GKs selected or the treatment of objects having/lacking the `last-application-configuration` annotation. - -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> Expand on adoption (taking over management of a set created by another tool) -> vs migration (taking over management of a set created by the old pruning mechanism) -> and how kubectl will or won't facilitate these flows. -> -> <<[/UNRESOLVED]>> +During implementation of the alpha we will explore to what extent we can +optimize this overlap discovery, particularly in conjunction with +server-side-apply which does not require an object read before applying. +A richer apply tooling than kubectl does will likely establish watches +on the objects before applying them, to monitor object health and status. +However, this is out of scope for kubectl and thus we will likely have to +optimize differently for kubectl. In the worst case, we will have to fetch +the objects before applying (with a set of label-filtered LIST requests), +we will explore to what extent that can be amortized over other kubectl +operations in alpha. One interesting option may be to use the fieldManager, +choosing a fieldManager that includes the applyset ID to automatically +detect conflicts (by _not_ specifying force); we intend to explore +how this looks in practice and whether other options present themselves. + +We differentiate between "adoption" (taking over management of a set of +objects created by another tool), vs "migration" (taking over management of +a set of objects created with the existing pruning mechanism). + +We will not support "adoption" of existing applysets initially, other than +by re-applying "over the top". Based on user feedback, we may require a flag +to adopt existing objects / applysets. + +In the alpha scope, we will explore suitable "migration" tooling for moving +from existing `--prune` objects. Note that migration is not trivial, in that +different users may expect different behaviors with regard to the GKs selected +or the treatment of objects having/lacking the `last-application-configuration` +annotation. We intend to create `migrate` as an explicit subcommand of `apply`, +rather than trying to overload the "normal flow" apply command. ### Tooling Interoperability @@ -617,15 +642,15 @@ For read operations, we expect that using different tooling shall generally be s For write operations, we need to be more careful. Deleting an applyset using the “wrong tool” should be safe, but we will likely include a confirmation if deleting an applyset using the “wrong tool”, particularly unknown tools. We expect that porcelain tools may define richer behavior on delete, so this is the equivalent of pulling the power cable on an applyset instead of performing a clean shutdown. -We do not believe that update operations are safe if using the “wrong tool”, because that tooling may have additional metadata that would then not be updated. Tooling should generally reject applying on top of unknown applysets. PorcelainTooling may choose to recognize other tooling and implement specific logic there; in particular this may be useful for moving between different major versions of the same tooling. We may implement a `--force` flag, but this would likely be logically equivalent in outcome to a full applyset deletion and recreation, though with the potential (but not the guarantee) to be less disruptive. - -> <<[UNRESOLVED @justinsb @KnVerey]>> -> -> How should we recognize that we’re using the “wrong’ tool? -> Should we have something like `applyset.k8s.io/tooling: helm/v2.0.6` on the applyset? -> -> <<[/UNRESOLVED]>> +We do not believe that update operations are safe if using the “wrong tool”, because that tooling may have additional metadata that would then not be updated. Tooling should generally reject applying on top of unknown applysets. Porcelain tooling may choose to recognize other tooling and implement specific logic there; in particular this may be useful for moving between different major versions of the same tooling. We may implement a `--force` flag, but this would likely be logically equivalent in outcome to a full applyset deletion and recreation, though with the potential (but not the guarantee) to be less disruptive. +In order to identify usage of the "wrong tool", we define a further annotation +`applyset.k8s.io/tooling`, which tooling can set to protect their applysets. +The value should be something like `kubectl/v1.27.3` or `helm/v2.0.6`, +i.e. `/`. Compatible porcelain tooling should recognize that +a different tool is managing the applyset and provide an appropriate warning. +We intend to explore the trade-off between safety and user-friendly behaviour +here, during evolution of the feature in alpha and beyond. ### Security Considerations @@ -710,6 +735,18 @@ We expect no non-infra related flakes in the last month as a GA graduation crite ### Graduation Criteria +We would like this functionality to replace the existing uses of `--prune`. We have +chosen to take an approach that is a better and supportable evolution of the existing +label based pruning, rather than a revolutionary new approach, to try to enable migration. + +At some point we might deprecate the existing `--prune` functionality, to encourage users +to migrate. A suitable timeline would probably be to begin deprecation at beta, and to +not remove the functionality until at least applyset reaches GA + 1 version. However, we +intend to gather feedback from early alphas here - in particular we want to discover: + +* Are there `--prune` use-cases we do not cover? +* Do existing `--prune` users migrate enthusiastically (without any "nudge" from deprecation)? + -- ``: `` - `` +- `k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/apply`: `2023-01-24` - `76.5%` +- `k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/diff`: `2023-01-24` - `33%` (and reportedly 0% for prune.go!) ##### Integration tests @@ -717,6 +756,8 @@ For Beta and GA, add links to added tests together with links to k8s-triage for https://storage.googleapis.com/k8s-triage/index.html --> +CLI tests will be added to both `test/cmd/diff.sh` and `test/cmd/apply.sh`. + - : ##### e2e tests From f8169d47f01076b4143d6569bc6fe4d55baeb429 Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Thu, 26 Jan 2023 16:34:33 -0500 Subject: [PATCH 08/19] Apply prune jan25 (#3) * More clearly delineate specification vs kubectl details * Move design details of spec to Design Details section --- .../3659-kubectl-apply-prune/README.md | 322 ++++++++++-------- 1 file changed, 186 insertions(+), 136 deletions(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index 28e0d9ccc2d..ec453bb031a 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -77,23 +77,44 @@ tags, and then generate with `hack/update-toc.sh`. - [Goals](#goals) - [Non-Goals](#non-goals) - [Background](#background) + - [Definitions](#definitions) - [Use case](#use-case) - [Feature history](#feature-history) - [Current implementation](#current-implementation) - [Problems with the current implementation](#problems-with-the-current-implementation) - [Correctness: object leakage](#correctness-object-leakage) + - [UX: flag changes affect correctness](#ux-flag-changes-affect-correctness) - [Scalability](#scalability) - [UX: easy to trigger inadvertent over-selection](#ux-easy-to-trigger-inadvertent-over-selection) - - [UX: flag changes affect correctness](#ux-flag-changes-affect-correctness) - [UX: difficult to use with custom resources](#ux-difficult-to-use-with-custom-resources) - [Sustainability: incompatibility with server-side apply](#sustainability-incompatibility-with-server-side-apply) + - [Related solutions in the ecosystem](#related-solutions-in-the-ecosystem) + - [Helm](#helm) + - [Carvel kapp](#carvel-kapp) + - [kpt](#kpt) + - [Google ConfigSync](#google-configsync) - [Proposal](#proposal) - [User Stories (Optional)](#user-stories-optional) - [Story 1](#story-1) - [Story 2](#story-2) - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) - [Risks and Mitigations](#risks-and-mitigations) -- [Design Details](#design-details) +- [Design Details: ApplySet Specification](#design-details-applyset-specification) + - [ApplySet Member Objects](#applyset-member-objects) + - [Labels](#labels) + - [ApplySet Parent Objects](#applyset-parent-objects) + - [Labels and annotations](#labels-and-annotations) + - [Optional "hint" annotations](#optional-hint-annotations) + - [Parent object management](#parent-object-management) + - [ApplySet scopes](#applyset-scopes) + - [Tooling Interoperability](#tooling-interoperability) + - [Objects with owner references](#objects-with-owner-references) + - [Versioning](#versioning) +- [Design Details: Kubectl Pruning](#design-details-kubectl-pruning) + - [Supported ApplySet Parent Kinds](#supported-applyset-parent-kinds) + - [Efficient Listing of ApplySet Contents](#efficient-listing-of-applyset-contents) + - [Kubectl Commands and Flags](#kubectl-commands-and-flags) + - [Security Considerations](#security-considerations) - [Test Plan](#test-plan) - [Prerequisite testing updates](#prerequisite-testing-updates) - [Unit tests](#unit-tests) @@ -112,6 +133,9 @@ tags, and then generate with `hack/update-toc.sh`. - [Implementation History](#implementation-history) - [Drawbacks](#drawbacks) - [Alternatives](#alternatives) + - [Full GKNN listing](#full-gknn-listing) + - [OwnerRefs](#ownerrefs) + - [ManagedFields](#managedfields) - [Infrastructure Needed (Optional)](#infrastructure-needed-optional) @@ -176,10 +200,7 @@ When creating objects with `kubectl apply`, it is frequently desired to make cha - MUST provide a way to accurately preview which objects will be deleted - MUST support namespaced and non-namespaced resources; SHOULD support them within the same operation - SHOULD use a low-level "plumbing" object grouping mechanism over which more sophisticated abstractions can be built by "porceline" tooling. - +- SHOULD allow for listing of grouping objects themselves ### Non-Goals @@ -364,63 +385,11 @@ Distinct sets of synchronized resources are represented by RootSync / RepoSyncs, ## Proposal -A v2-prunable "apply set" is associated with an object on the cluster. We define a set of standardized labels and annotations that identify the “parent object” of the apply set and the “member objects” of that parent. We operate at the plumbing layer; we aim to support: - -- listing the parent objects efficiently (porcelain may expose this as listing groups of objects as managed by the various tools) -- listing the member objects for a specific parent object efficiently (porcelain may use this for advanced diffing and pruning, and for presenting objects grouped by their higher-level aggregation) -- basic apply-with-prune operations, where it creates or reuses a Secret in the cluster as the parent object. - -### Apply Set - -"Apply set" refers to a group of resources that are applied to the cluster by a tool. An apply set has a “parent” object of the tool’s preference. This “parent” object can be implemented using a ConfigMap, Secret, or a CRD of the tool’s choice. - -“ApplySet” is used to refer to the parent object in this design document, though the actual concrete resource on the cluster will typically be of a different Kind. We might think of ApplySet as a “duck-type” based on the `applyset.k8s.io/id` label proposed here. - -### Member Object Labels - -Objects that are part of an ApplySet should carry a standardized label, with a key of: - -```yaml -applyset.k8s.io/part-of: -``` - -The `` can be chosen essentially arbitrarily (subject to the limits of label values). - -We recommend that tooling uses a prefix that is its "tool name", e.g. `kubectl.` or `helm.`, -followed by some tooling-specific unique value. We can investigate maintaining a formal registry, -but we initially reserve the following prefixes: - -* `kubectl.` for kubectl -* `helm.` for helm -* `kpt.` for kpt -* `uid.` for where the suffix is the UID of the applyset object (e.g. `uid.e049464e-4583-4642-9649-93dcb0e96bd4`) -* `id.` for where the suffix is the group-kind followed by the namespace followed by the name, such as - `id.configmaps.ns1.parent1`. (While this is a little tricky to parse, it should be unique because - neither namespaces nor object names allows dots.) - -### ApplySet Object Labels - -ApplySets should also have a “parent object” in the cluster. This ApplySet object can (in theory) be of any type. For performance reasons we later propose limiting to ConfigMap, Secret and custom resources with a specific label. In future we may define a common CRD, but we believe we can achieve a reasonable user experience without defining one. Many existing tools avoid using a CRD, so that they can be used by people without the cluster-admin permissions needed to install a CRD. (This also avoids CRD versioning problems etc). - -The ApplySet object should be labeled with: - -```yaml -applyset.k8s.io/id: -``` - -Implicit in this are a few assumptions: - -- An object can be part of at most one ApplySet. This is a limitation, but seems to be a good one in that objects that are part of multiple ApplySets are complicated both conceptually for users and in terms of implementation behaviour. -- An ApplySet object can be part of another ApplySet (sub-ApplySets). - -How the ApplySet object is specified is a tooling decision. Gitops based tooling may choose to make the ApplySet object explicit in the source git repo. Other tooling may choose to leverage their existing concepts, for example mapping to a Secret or ConfigMap that they are creating already. The tooling is responsible for consistently specifying the ApplySet object across apply invocations, so that pruning can be done consistently. - - -For kubectl specifically, we propose supporting but not requiring explicitly -provided parent objects, with automatic object creation in the latter case. -This is explained in more detail below. +A v2-prunable "apply set" is associated with an object on the cluster. We define a set of standardized labels and annotations that identify the “parent object” of the apply set and the “member objects” of that parent. This specification forms a plumbing layer upon which multiple tools can build their own implementations of set-based operations such as pruning. +The specification aims to be very lightweight, so that it is as easy as possible for tools with their own existing grouping mechanisms to opt in for greater interoperability. By using the standardized labels proposed here, tools can interoperate with other tooling and enjoy protection against their resources being changed by another tool (such as kubectl). Tooling is not required to implement these labels, and we are not introducing new behaviours for “unenlightened” objects. +Under [Design Details: ApplySet Specification](#design-details-applyset-specification), we set out this label-based design, which is capable of encompassing the object groupings that kubectl and other tools need while avoiding the pitfalls explained in the background section. Under [Design Details: Kubectl Pruning](#design-details-kubectl-pruning), we explain how this specification can be used by `kubectl apply` to achieve the primary goal of this KEP: fixing the existing pruning functionality without turning kubectl into a "porceline" tool itself. ### User Stories (Optional) @@ -458,94 +427,139 @@ How will UX be reviewed, and by whom? Consider including folks who also work outside the SIG or subproject. --> -## Design Details -### Efficient Listing of ApplySets +## Design Details: ApplySet Specification -In order to support listing all the applysets in theory we would need to query all GKs with a label selector. However, we can reduce the set of GKs that need to be queried with two optimizations: +"Apply set" refers to a group of resources that are applied to the cluster by a tool. An apply set has a “parent” object of the tool’s preference. This “parent” object can be implemented using a Kind of the tool’s choice. -* For built-in types, we limit to ConfigMaps and Secrets. +“ApplySet” is used to refer to the parent object in this design document, though the actual concrete resource on the cluster will typically be of a different Kind. We might think of ApplySet as a “duck-type” based on the `applyset.k8s.io/id` label proposed here. -* For custom resources, we require that CRDs that define types that can be used as ApplySet objects be labeled with a label with a name of `applyset.k8s.io/role/applyset`. +Implicit in this proposal are a few assumptions: -The value is currently ignored, but implementors should set an empty value to -be forwards-compatible with future evolution of this convention. +- An object can be part of at most one ApplySet. This is a limitation, but seems to be a good one in that objects that are part of multiple ApplySets are complicated both conceptually for users and in terms of implementation behaviour. +- An ApplySet object can be part of another ApplySet (sub-ApplySets). -A `kubectl apply list-applysets -n ns` command would therefore do the following queries: +### ApplySet Member Objects -```bash -kubectl get secret -n ns -l applyset.k8s.io/id # --only-partial-object-metadata -kubectl get configmap -n ns -l applyset.k8s.io/id # --only-partial-object-metadata +#### Labels -for crd in $(kubectl get crd -l applyset.k8s.io/role/applyset); do -kubectl get $crd -n ns -l applyset.k8s.io/id # --only-partial-object-metadata -done +Objects that are part of an ApplySet MUST carry two standardized labels: + +```yaml +applyset.k8s.io/part-of: # REQUIRED +applyset.k8s.io/controller-ref: # REQUIRED ``` -Optimizations are possible here. For example we can likely cache the list of CRDs. However, while the number of management tools may grow, the number of management ecosystems is relatively small, and we would expect a given cluster to use only a fraction of the management ecosystems. So the number of queries here is likely to be small. Moreover these queries can be executed in parallel, we can now rely on priority-and-fairness to throttle these appropriately without needing to self-throttle client-side. +The `applyset.k8s.io/part-of` is the source of truth for membership in a set. Its `` value can be chosen essentially arbitrarily (subject to the limits of label values), but MUST match the value of `applyset.k8s.io/id` on the parent (see below). As such, sourcing this value from the end user is recommended. -In future, we may define additional “index” mechanisms here to further optimize this (controllers or webhooks that watch these labels and populate an annotation on the namespace, or support in kube-apiserver for cross-object querying). However the belief is that this is likely not needed at the current time. +The `applyset.k8s.io/controller-ref` is used to detect collisions between applysets with the same name, which is a bad practice that cannot be reliably prevented. Its `` value will be in a machine-readible format dictated by this specification, likely either the parent's UUID or a base64 encoding of its GKNN (to be finalized during initial prototyping). Tooling that manages objects MUST set this label, and MUST verify that the identifier on the member object matches the parent before taking any action (e.g. deletion) on that member. -### Efficient Listing of ApplySet Contents +### ApplySet Parent Objects -We want to support efficient listing of the objects that belong to a particular applyset. In theory, this again requires the all-GK listing (with a label filter). An advantage of this approach is that this remains an option: as we implement optimizations we may also periodically run a “garbage collector” to verify that our optimizations have not leaked objects, perhaps `kubectl apply fsck` or a plugin. +ApplySet parent objects can (in theory) be of any type, though specific tools may limit the number of types they support for performance or UX reasons. While a purpose-made cluster-scoped CRD would be a logical choice, the specification has no opinion on this, so as to accommodate the many and various choices existing tools have already made. In choosing types to support as parents, tools should consider what permissions their target users typically have on their clusters; for instance, they may not have permissions to install CRDs. -We already know the label selector for a given applyset, by convention: we take the id from the value of the `applyset.k8s.io/id` label, and that becomes the required value of the `applyset.k8s.io/part-of` label. +#### Labels and annotations -In order to narrow the list of GKs, we require the applyset object to define the list of GKs in use. The plumbing tooling can optimize selection of the objects in this applyset based on this list. +ApplySets MUST also have a “parent object” in the cluster. The ApplySet object MUST be labeled with: -“Porcelain” tooling can still perform tooling-specific GK identification. Tooling generally can use their existing mechanisms, be they more efficient or more powerful or just easier to continue to support. However, by using the standardized labels proposed here, they can interoperate with other tooling and enjoy protection against their resources being changed by another tool (such as kubectl). Tooling is not required to implement these labels, and we are not introducing new behaviours for “unenlightened” objects. +```yaml +applyset.k8s.io/id: # REQUIRED +``` -To identify the GKs in use, kubectl and applyset-compatible tooling shall add an annotation `applyset.k8s.io/contains-group-kinds`; we use an annotation instead of a label because the annotation can be larger, and because we do not need to select on this value. The value of the annotation shall be a comma separated list of the group-kinds, in the fully-qualified name format, i.e. `.`. An example annotation value might therefore look like: `certificates.cert-manager.io,configmaps,deployments.apps,secrets,services`. Note that we do not include the version; formatting is a different concern from “applyset membership”. +The `applyset.k8s.io/id` label is what makes the object an ApplySet parent object. As mentioned above, its `` value is replicated on set members' `applyset.k8s.io/part-of` labels, and its value can be chosen arbitrarily (subject to the limits of label values). As this value is likely to be provided by end users to identify the set they intend to operate on, allowing them to choose a value meaningful to them is recommended. -To avoid spurious updates and conflicts, the list must be sorted alphabetically. The list may include GKs where there are no resources actually labeled with the applyset-id, but to avoid churn this should be avoided and ideally only be a transitional step during an apply or prune operation. +Additionally, ApplySet parents MUST be labelled with: -We may in future define additional mechanisms, such as supporting a field -selector on the CRD that identifies a strongly-typed list; we do not plan -to do this in the alpha. +```yaml +applyset.k8s.io/tooling: # REQUIRED +``` -Where no list of GKs can be determined the tooling should warn that we are performing a full-GK scan. As discussed in the interoperability section, tooling should not populate the annotation unless it believes itself to be the manager of an applyset. +The value should be something like `kubectl/v1.27` or `helm/v3` or `kpt/v1.0.0`, +i.e. `/`, and tooling should refuse to mutate ApplySets belonging to other tools. +For more background and guidance on this topic, see the [interoperability](#tooling-interoperability) section. -In pseudo-code, to discover the existing members of an applyset: +ApplySets MAY have an annotation extending their scope, which MUST be respected by all tools if present: -```bash -for-each gk in $(split group-kind-annotation); do -kubectl get $gk -n ns -l applyset.k8s.io/id # --only-partial-object-metadata -done +```yaml +applyset.k8s.io/additional-namespaces: [,] # OPTIONAL +``` + +The `applyset.k8s.io/additional-namespaces` annotation extends the scope of the ApplySet. By default, the scope of an ApplySet is its parent object's namespace. When the parent is cluster-scoped but refers to namespace kinds, or when the set spans multiple namespaces (which is not recommended, but allowed), this annotation an be used to extend the ApplySet's scoped to the listed namespaces. As cross-namespace ApplySets are not particularly encouraged, we do not currently optimize this further. In particular, we do not specify the GKs per namespace. We can add more annotations in future should the need arise. + +The value of this annotation is a comma-separated list of the names of the namespaces (other than the ApplySet namespace) in which objects are found, for example `default,kube-system,ns1,ns2`. We reserve the empty value. As with `applyset.k8s.io/contains-group-kinds`, this list of namespaces must be sorted alphabetically, and should be minimal (other than transiently during applyset mutations). + +#### Optional "hint" annotations + +The ApplySet parent object MAY also have one or more of the following annotations that help tooling identify ApplySet members more efficiently. These are annotations instead of labels because annotations can be larger, and because we definitely do not need to select on these values. + +While the use of either of these annotations assists arbitrary tools in listing the applyset members, their use is not required by the specification, and tooling should not populate such annotations unless it believes itself to be the manager of an applyset. + +Most tools will likely want to have some GK/object identification mechanism along these lines for performance and permissions reasons, but “porcelain” tooling can continue to do using its existing mechanisms, be they more efficient or more powerful or just easier to continue to support. + +When a tool that wants to list the members of an applyset and cannot determine the list of GKs (i.e. because neither hint annotation is used), it may support “discovery”, likely warning that a full cluster crawl is being attempted. Insufficient permissions errors are likely with such functionality, and when they are encountered, the tool should warn that the membership list may be incomplete. + +If feedback shows a need for it, we may in future define additional "hint" mechanisms, such as supporting a field selector on the CRD that identifies a strongly-typed list of member objects. + +```yaml +applyset.k8s.io/contains-group-kinds: .[,]` # OPTIONAL +``` + +The `applyset.k8s.io/contains-group-kinds` annotation is an optional "hint" annotation tools can populate and use to optimize listing of member objects. Tooling not using this annotation may safely ignore it. Since the annotation on the member objects themselves remains the source of truth for set membership, tools making use of this optimization should consider also providing or periodiacally automating a resync of the hint annotation. + +When present, the value of this annotation shall be a comma separated list of the group-kinds, in the fully-qualified name format, i.e. `.`. An example annotation value might therefore look like: `certificates.cert-manager.io,configmaps,deployments.apps,secrets,services`. Note that we do not include the version; formatting is a different concern from “applyset membership”. + +To avoid spurious updates and conflicts, the list must be sorted alphabetically. The list may include GKs where there are no resources actually labeled with the applyset-id, but to avoid churn this should be avoided and ideally only be a transitional step during applyset mutations. + +If the list in this annotation includes namespaced-scoped GKs on a cluster-scoped parent with no `applyset.k8s.io/additional-namespaces` annotation, the tooling should output an error. + +```yaml +applyset.k8s.io/inventory: ./.[,] # OPTIONAL ``` -### Cluster-scoped ApplySets +The `applyset.k8s.io/inventory` annotation is an alternative optional "hint" annotation tools can populate and use to optimize listing of member objects. Tooling not using this annotation may safely ignore it. Since the annotation on the member objects themselves remains the source of truth for set membership, tools making use of this optimization should consider also providing or periodically automating a resync of the hint annotation. + +When used, its value must be a comma separated list of all the GKNNs in use in the ApplySet. To avoid spurious updates and conflicts, the list must be sorted alphabetically. + +Tooling using this annotation should take care to ensure that all listed GKNNs are in fact valid members of the ApplySet based on its scope. For instance, a cluster-scoped parent without a `applyset.k8s.io/additional-namespaces` annotation cannot reference namespace-scoped GKNNs, and a namespace-scoped parent without that annotation cannot reference GKNNs in other namespaces. -We need to support ApplySets that are cluster-scoped, for example ApplySets that include installation of CRDs (such as cert-manager). The mechanisms we have defined here work for cluster-scoped ApplySets. Today’s tooling will create an managing object in a namespace, and likely a cluster-scoped CRD would be more intuitive than a namespace-scoped resource. However, no additional explicit support for cluster-scoped ApplySets is required or proposed at the current time (but cf the next section for cross-namespace considerations). +To remain compliant with the specification, tools using this particular annotation should still refrain from operating on (e.g. deleting) a member object before verifying its `applyset.k8s.io/part-of` and `applyset.k8s.io/controller-ref` annotations. -### Cross-Namespace ApplySet Contents +#### Parent object management -When querying for ApplySet contents, an ApplySet could contain cluster-scoped resources or could contain resources in other namespaces. Querying for this content is generally going to require more permissions and be slower, so we would like to avoid over-querying here. +How the ApplySet object is specified is a tooling decision. Gitops based tooling may choose to make the ApplySet object explicit in the source git repo. Other tooling may choose to leverage their existing concepts, for example mapping to a Secret or ConfigMap that they are creating already. The tooling is responsible for consistently specifying the ApplySet object across apply invocations, so that pruning can be done consistently. -Best practice is likely to avoid ApplySets spanning namespaces. However, sometimes this is unavoidable - particularly when managing cluster-scoped objects - and the “plumbing” tooling cannot enforce this restriction. +### ApplySet scopes -Where a GK is known to be part of the ApplySet and is cluster-scoped, we should naturally query for those objects at cluster scope; any permission problems here should be surfaced as errors. When a tool cannot determine the list of GKs for an ApplySet, it may support “discovery”, likely warning that the ApplySet does not define a list of GKs, and then attempt to perform cluster-scoped queries, likely warning if there are insufficient permissions. +Although the best practice is generally to constrain ApplySets to a single scope where possible, sometimes multi-scoped sets are unavoidable in the real world. Therefore, the mechanisms we have defined here allow for ApplySets that are cluster-scoped, multi-namespace or mixed-scoped (for example ApplySets that include installation of CRDs such as cert-manager). -For the alpha scope, this functionality will be restricted to subcommands of apply -(like the migrate functionality). We will likely want a command similar in spirit -to `fsck`. We will add warnings/suggestions to the main "apply" flow when we detect -problems that might require a full-scan / discovery. We may extend this based on -user-feedback from the alpha. +If the parent object is namespaced, member objects may be in that same namespace or at the cluster scope. The `applyset.k8s.io/additional-namespaces` annotation can be used to allow members in additional namespaces. This is purely additive; it is not possible to create a namespaced parent object that excludes its own namespace. -For GKs that are namespace-scoped, we would normally expect those to be part of an ApplySet object in the same namespace. We define an additional annotation however for cross-namespace ApplySets: `applyset.k8s.io/additional-namespaces`. The value of this annotation is a comma-separated list of the names of the namespaces (other than the ApplySet namespace) in which objects are found, for example `default,kube-system,ns1,ns2`. Note that there is no need to include this specifically for cluster-scoped objects, as those are covered by the group-kind list. We reserve the empty value. If this annotation is present, the tooling will query namespace-scoped resources in those namespaces in addition to the namespace of the ApplySet object (if any). If this annotation is not present on a namespace-scoped ApplySet parent object, the tooling will query namespace-scoped resources only in the same namespace as the ApplySet parent object. If the annotation is not present on a cluster-scoped ApplySet parent object, the tooling will not query namespace-scoped resources at all and should output an error if given namespace-scoped GKs. +If the parent object is cluster-scoped, member objects by default are at the cluster scope. The `applyset.k8s.io/additional-namespaces` annotation can be used to allow member objects in one or more namespaces. -As with `applyset.k8s.io/contains-group-kinds`, this list of namespaces must be sorted alphabetically, and should be minimal (ideally other than during apply and prune operations). +``` +<<[UNRESOLVED @justinsb ]>> -As cross-namespace ApplySets are not particularly encouraged, we do not currently optimize this further. In particular, we do not specify the GKs per namespace. We can add more annotations in future should the need arise. +It is not possible to prevent an ApplySet from referring to cluster-scoped resources. Should this instead be explicitly opt-in, like cross-namespace is? That could encourage best practices and improve performance by reducing the default scope. -Where an applyset includes both cluster-scoped and namespace-scoped resources, -by reducing to the above cases. The set of relevant resources is determined -by consulting the `applyset.k8s.io/contains-group-kinds` annotation; whether -those kinds are cluster-scoped or namespace-scoped are found using the normal -API discovery mechanisms. Cluster-scoped resources ignore the -`applyset.k8s.io/additional-namespaces` annotation, namespace-scoped resources -combine the current namespace from the applyset object (if that is itself -namespace-scoped) with the namespaces from the annotation. +<<[/UNRESOLVED]>> +``` + +### Tooling Interoperability + +There is a rich ecosystem of existing tooling that we hope will adopt these labels and annotations. So that different tooling can interoperate smoothly, we define some requirements for safe interoperability here. + +For read operations, we expect that using different tooling shall generally be safe. As these labels do not collide with existing tooling, we would expect that objects installed with existing tooling would be invisible to the porcelain tooling until they had been updated to include the labels. We do not propose to implement “bridges” to existing tooling, rather as the proposal here is lightweight and small, it makes more sense to update the existing tooling. We may add warnings such as “applysets using an old version of X detected, upgrade to v123 of X to work with those applysets”. + +For write operations, we need to be more careful. Deleting an applyset using the “wrong tool” should be safe, but we will likely include a confirmation if deleting an applyset using the “wrong tool”, particularly unknown tools. We expect that porcelain tools may define richer behavior on delete, so this is the equivalent of pulling the power cable on an applyset instead of performing a clean shutdown. + +We do not believe that update operations are safe if using the “wrong tool”, because that tooling may have additional metadata that would then not be updated. Tooling should generally reject applying on top of unknown applysets. Porcelain tooling may choose to recognize other tooling and implement specific logic there; in particular this may be useful for moving between different major versions of the same tooling. + +In order to identify usage of the "wrong tool", we rely on the `applyset.k8s.io/tooling` annotation, +which tooling can set to protect their applysets. +Specification-compliant porcelain tooling MUST recognize that +a different tool is managing the applyset and provide an appropriate error or warning. +We intend to explore the trade-off between safety and user-friendly behaviour +here, during evolution of the feature in alpha and beyond. ### Objects with owner references @@ -571,9 +585,60 @@ include a `:` character, so if we needed to version a label we can use `v2.`, however our usage of labels is primarily around matching opaque applyset-id tokens and thus seems unlikely to need versioning. -### Kubectl Reference Implementation +## Design Details: Kubectl Pruning + +This KEP describes both a lightweight specification and a way to use that specification as the machinery backing an improved `kubectl apply --prune`. The specification itself is described in the [ApplySet Specification](#design-details-applyset-specification) section. This section focuses on how it will be put to use in kubectl. + +### Supported ApplySet Parent Kinds + +While the ApplySet specification itself does not restrict the kinds that can be used as parent objects, existing tools typically allow a small set of options in practice. For kubectl, we propose initially supporting Secret, ConfigMap, and specially configured custom resources. + +For custom resources, we require that CRDs that define types that can be used as ApplySet objects be labeled with a label with a name of `applyset.k8s.io/role/applyset`. The value is currently ignored, but implementors should set an empty value to be forwards-compatible with future evolution of this convention. + +This proposed restriction on supported Kinds is both for simplicity, and in anticipation of a performance optimization for the `kubectl apply list-apply-sets` command that we are considering for the beta timeline. Namely, this restriction significantly reduces the number of API resources the implementation of that command would need to call. A `kubectl apply list-applysets -n ns` command would therefore do the following queries: + +```bash +kubectl get secret -n ns -l applyset.k8s.io/id # --only-partial-object-metadata +kubectl get configmap -n ns -l applyset.k8s.io/id # --only-partial-object-metadata + +for crd in $(kubectl get crd -l applyset.k8s.io/role/applyset); do +kubectl get $crd -n ns -l applyset.k8s.io/id # --only-partial-object-metadata +done +``` -We will develop a reference implementation of this specification in kubectl, with the intention of providing a supportable replacement for the current alpha `kubectl apply --prune` semantics. Our intention is not to change the behavior of the existing `--prune` functionality, but rather to produce an alternative that users will happily and safely move to. We can likely trigger the V2-semantics when the user specifies an applyset flag, so that this is intuitive and does not break existing prune users. The proposal may evolve at the coding/PR stage, but the current plan is as follows. Overview of CLI proposed follows. +Optimizations are possible here. For example we can likely cache the list of CRDs. However, while the number of management tools may grow, the number of management ecosystems is relatively small, and we would expect a given cluster to use only a fraction of the management ecosystems. So the number of queries here is likely to be small. Moreover these queries can be executed in parallel, we can now rely on priority-and-fairness to throttle these appropriately without needing to self-throttle client-side. + +In future, we may define additional “index” mechanisms here to further optimize this (controllers or webhooks that watch these labels and populate an annotation on the namespace, or support in kube-apiserver for cross-object querying). However the belief is that this is likely not needed at the current time. + +A drawback of this approach is that a `list-apply-sets` command operates directly on the plumbing layer, and appears to be listing _all_ ApplySets from any tool, not just those kubectl created for pruning purposes. It is possible this will be misleading, as other tools may have used kinds beyond this restriction (they could do so even if the specification advised against it, which it currently does not). + +We may relax the restriction on supported Kinds in the future based on user feedback or a decision not to implement the `list-apply-sets` command (or a decision not to optimize it for performance over inclusiveness). + +### Efficient Listing of ApplySet Contents + +We want to support efficient listing of the objects that belong to a particular applyset. In theory, this again requires the all-GK listing (with a label filter). An advantage of this approach is that this remains an option: as we implement optimizations we may also periodically run a “garbage collector” to verify that our optimizations have not leaked objects, perhaps `kubectl apply fsck` or a plugin. + +We already know the label selector for a given applyset, by convention: we take the id from the value of the `applyset.k8s.io/id` label, and that becomes the required value of the `applyset.k8s.io/part-of` label. + +In order to narrow the list of GKs, kubectl will use the optional `applyset.k8s.io/contains-group-kinds` annotation described in the [optional parent object annotations](#optional-annotations) section to store the list of GKs in use. Whether those kinds are cluster-scoped or namespace-scoped are found using the normal API discovery mechanisms. + +In pseudo-code, to discover the existing members of an applyset: + +```bash +for-each gk in $(split group-kind-annotation); do +kubectl get $gk -n ns -l applyset.k8s.io/id # --only-partial-object-metadata +done +``` + +If the `applyset.k8s.io/additional-namespaces` annotation is present, any namespaced queries will need to be repeated for each target namespace. + +If the contains-group-kinds annotation is missing, kubectl will initially consider this an error. Based on feedback, we can consider either falling back on a (very slow) full-GK scan to populate the annotation (after confirming kubectl owns the parent), or pointing users to a separate command (similar in spirit to `fsck`) that will do so. We will add warnings/suggestions to the main "apply" flow when we detect problems that might require a full-scan / discovery. We may extend this based on user-feedback from the alpha. + +Based on performance feedback, we can also consider switching to the alternative `applyset.k8s.io/inventory` hint annotation. Even if we do not trust the GKNN list for deletion purposes (we cannot, as it is not the source of truth), it could be used to optimize certain specific cases, most notably the no-op scenario where the current set exactly matches the list. + +### Kubectl Commands and Flags + +The intention of the proposed changes is to provide a supportable replacement for the current alpha `kubectl apply --prune` semantics. Our intention is not to change the behavior of the existing `--prune` functionality, but rather to produce an alternative that users will happily and safely move to. We can likely trigger the V2-semantics when the user specifies an applyset flag, so that this is intuitive and does not break existing prune users. The proposal may evolve at the coding/PR stage, but the current plan is as follows. Required for an MVP release: - `KUBECTL_APPLYSET_ALPHA=1` environment variable: Required to expose the new flags/commands during alpha. @@ -583,7 +648,7 @@ Required for an MVP release: Tentatively proposed for future iterations (more specific design details to follow after MVP): - `kubectl apply generate-apply-set --selector=[key=val] --legacy-allow-list=[]`: command to migrate from the legacy pruning system to this new one. -- `kubectl apply verify-apply-set `: `fsck`-style functionality to update the annotations on the parent applyset objects. +- `kubectl apply verify-apply-set [--fix]`: `fsck`-style functionality to update the annotations on the parent applyset objects. - `kubectl apply view-apply-set -o name|json|yaml`: A command for viewing applyset membership, ideally in a way that can be programmatically chained. - `kubectl apply disband-apply-set `: removes the `applyset.k8s.io/id` from all members and then deletes the parent applyset object. - `kubectl apply list-apply-sets`: view apply sets, including those managed by other tools. @@ -627,7 +692,7 @@ kubectl apply list-objects -n ns1 –applyset=set1 We intend to treat the flag and any subcommands as alpha commands initially. During alpha, users will need to set an environment variable (e.g. KUBECTL_APPLYSET_ALPHA) to make the flag available. -Commands will verify that the value of `applyset.k8s.io/tooling` has the `kubectl/` prefix before making any mutation, failing with an error if the annotation is present with any other value. It will set this label to `kubectl/vX.XX` (e.g. kubectl/v1.27) when creating/adopting resources as parent objects and update the semver as needed. At least initially, a missing tooling label or blank label value will also be considered an error, though this is not strictly required by the proposed spec and could be relaxed in the future. +Commands will verify that the value of `applyset.k8s.io/tooling` has the `kubectl/` prefix before making any mutation, failing with an error if the annotation is present with any other value. It will set this label to `kubectl/vX.XX` (e.g. kubectl/v1.27) when creating/adopting resources as parent objects and update the semver as needed. At least initially, a missing tooling label or blank label value will also be considered an error, though this is not strictly required by the proposed spec and could be relaxed in the future. We may implement a `--force` flag, but this would likely be logically equivalent in outcome to a full applyset deletion and recreation, though with the potential (but not the guarantee) to be less disruptive. When `--apply-set=` is used (with no GVR), kubectl will automatically default the GVR to "secret", and will use server-side apply to create or update a Secret by that name in the targeted namespace, with the labels/annotations described here. If no namespace is specified, this is an error. Secret creation will happen at the beginning of the pruning phase rather than during the main apply operation. Server-side apply (SSA) will be used to create the Secret even if the main operation used client-side apply, and conflict forcing will be disabled regardless of its status on the main operation. Taking over an existing Secret is allowed, as long as it does not have any conflicting fields (no special criteria vs subsequent operations). @@ -672,23 +737,6 @@ annotation. We intend to create an explicit migration subcommand on `apply`, e. `kubectl apply generate-apply-set --selector=[key=val] --legacy-allow-list=[]`, rather than trying to overload the "normal flow" apply command. -### Tooling Interoperability - -There is a rich ecosystem of existing tooling that we hope will adopt these labels and annotations. So that different tooling can interoperate smoothly, we define some requirements for safe interoperability here. - -For read operations, we expect that using different tooling shall generally be safe. As these labels do not collide with existing tooling, we would expect that objects installed with existing tooling would be invisible to the porcelain tooling until they had been updated to include the labels. We do not propose to implement “bridges” to existing tooling, rather as the proposal here is lightweight and small, it makes more sense to update the existing tooling. We may add warnings such as “applysets using an old version of X detected, upgrade to v123 of X to work with those applysets”. - -For write operations, we need to be more careful. Deleting an applyset using the “wrong tool” should be safe, but we will likely include a confirmation if deleting an applyset using the “wrong tool”, particularly unknown tools. We expect that porcelain tools may define richer behavior on delete, so this is the equivalent of pulling the power cable on an applyset instead of performing a clean shutdown. - -We do not believe that update operations are safe if using the “wrong tool”, because that tooling may have additional metadata that would then not be updated. Tooling should generally reject applying on top of unknown applysets. Porcelain tooling may choose to recognize other tooling and implement specific logic there; in particular this may be useful for moving between different major versions of the same tooling. We may implement a `--force` flag, but this would likely be logically equivalent in outcome to a full applyset deletion and recreation, though with the potential (but not the guarantee) to be less disruptive. - -In order to identify usage of the "wrong tool", we define a further annotation -`applyset.k8s.io/tooling`, which tooling can set to protect their applysets. -The value should be something like `kubectl/v1.27` or `helm/v3` or `kpt/v1.0.0`, -i.e. `/`. Compatible porcelain tooling should recognize that -a different tool is managing the applyset and provide an appropriate warning. -We intend to explore the trade-off between safety and user-friendly behaviour -here, during evolution of the feature in alpha and beyond. ### Security Considerations @@ -1238,6 +1286,8 @@ Instead of encoding a list of GKs to scope in, we could encode a the full list o The reason it does not optimize performance in practice is that we're considering the source of truth for membership to be the `part-of` annotations on the resources themselves. This is useful for visibility and for ownership conflict avoidance, but it means we must retrieve the objects themselves to check the source of truth rather than relying on the GVKNN. Since individual GET calls are far more expensive than LISTs in the common case for pruning, in practice, we would end up extracting the GK list from any GKNN list and make the same calls we would have with just a GK list. If it is deemed worthwhile, we could indeed do this, and it would allow an additional layer of in-band drift detection via comparison of the precise list to the set of current labelled resources. +That said, the GKNN approach could likely be used to increase efficiency in a particularly common scenario: recognition that the set has not changed. We could choose to trust the listing in this scenario to avoid making any queries at all. A standard annotation for storing GKNN information is already part of this proposal, and we could switch the kubectl implementation to it based on experience with the alpha if desired. + Alternatively, we could omit the `part-of` label entirely (which leaves no means of ownership conflict management), or consider the GKNN list the source of truth (which leaves a much wider vector for object leakage in practice than GK listing does, in our opinion). ### OwnerRefs From 16ea122c5be394c4f23f74b1c7edbc0837955388 Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Thu, 26 Jan 2023 17:55:37 -0500 Subject: [PATCH 09/19] Updates from synchronous conversation --- .../3659-kubectl-apply-prune/README.md | 162 +++++++++--------- 1 file changed, 82 insertions(+), 80 deletions(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index ec453bb031a..427e46eba0f 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -100,6 +100,7 @@ tags, and then generate with `hack/update-toc.sh`. - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details: ApplySet Specification](#design-details-applyset-specification) + - [ApplySet Naming](#applyset-naming) - [ApplySet Member Objects](#applyset-member-objects) - [Labels](#labels) - [ApplySet Parent Objects](#applyset-parent-objects) @@ -199,7 +200,7 @@ When creating objects with `kubectl apply`, it is frequently desired to make cha - MUST natively support custom resources - MUST provide a way to accurately preview which objects will be deleted - MUST support namespaced and non-namespaced resources; SHOULD support them within the same operation -- SHOULD use a low-level "plumbing" object grouping mechanism over which more sophisticated abstractions can be built by "porceline" tooling. +- SHOULD use a low-level "plumbing" object grouping mechanism over which more sophisticated abstractions can be built by "porcelain" tooling. - SHOULD allow for listing of grouping objects themselves ### Non-Goals @@ -279,7 +280,7 @@ At Kubecon NA 2022, @seans3 and @KnVerey led a session discussing the limitation - Any changes to existing behavior are likely to break existing users. - Although `--prune` is technically in alpha, breaking existing workflows is likely to be unpopular. If the new solution is independent of the existing alpha, the alpha will need to be deprecated using a beta (at minimum) timeline, given how long it has existed. - There are several solutions in the community that have broadly evolved to follow the label pattern, and typically store the label and the list of GVKs on a parent object. Some solutions store a complete list of objects. - We could likely standardize and support the existing approaches, so that they could be more interoperable. kubernetes would define the “plumbing” layer, and leave nice user-facing “porcelain” to tooling such as helm. -- By defining a common plumbing layer, tools such as kubectl could list existing “applysets”, regardless of the tooling used to install them. +- By defining a common plumbing layer, tools such as kubectl could list existing “ApplySets”, regardless of the tooling used to install them. - `kubectl apply --prune` could use this plumbing layer as a new pruning implementation that would address many of the existing challenges, but would also simplify adoption of tooling such as Helm or Carvel. ### Current implementation @@ -389,7 +390,7 @@ A v2-prunable "apply set" is associated with an object on the cluster. We define The specification aims to be very lightweight, so that it is as easy as possible for tools with their own existing grouping mechanisms to opt in for greater interoperability. By using the standardized labels proposed here, tools can interoperate with other tooling and enjoy protection against their resources being changed by another tool (such as kubectl). Tooling is not required to implement these labels, and we are not introducing new behaviours for “unenlightened” objects. -Under [Design Details: ApplySet Specification](#design-details-applyset-specification), we set out this label-based design, which is capable of encompassing the object groupings that kubectl and other tools need while avoiding the pitfalls explained in the background section. Under [Design Details: Kubectl Pruning](#design-details-kubectl-pruning), we explain how this specification can be used by `kubectl apply` to achieve the primary goal of this KEP: fixing the existing pruning functionality without turning kubectl into a "porceline" tool itself. +Under [Design Details: ApplySet Specification](#design-details-applyset-specification), we set out this label-based design, which is capable of encompassing the object groupings that kubectl and other tools need while avoiding the pitfalls explained in the background section. Under [Design Details: Kubectl Pruning](#design-details-kubectl-pruning), we explain how this specification can be used by `kubectl apply` to achieve the primary goal of this KEP: fixing the existing pruning functionality without turning kubectl into a "porcelain" tool itself. ### User Stories (Optional) @@ -439,24 +440,39 @@ Implicit in this proposal are a few assumptions: - An object can be part of at most one ApplySet. This is a limitation, but seems to be a good one in that objects that are part of multiple ApplySets are complicated both conceptually for users and in terms of implementation behaviour. - An ApplySet object can be part of another ApplySet (sub-ApplySets). +### ApplySet Naming + +Each ApplySet MUST have an ID that can be used to uniquely identify the parent and member objects via the label selector conventions outlined in the following sections. As such, the name: +* is subject to the normal limits of label values +* MUST be generated in a way that provides reasonable uniqueness guarantees at the cluster scope + +This uniqueness constraint is designed to prevent over-selection due to naming collisions within overlapping scopes. To comply with it, this specification recommends that tooling choose IDs derived from identifiers that are already globally unique. For instance, the parent object's UUID or a base64 encoding of its GKNN would be suitable. We reserve the right to dictate a specific format at some point before this KEP goes into beta. + +This name does not need to be used for the `metadata.name` of the parent object. In fact, it is likely desirable for tooling to allow end users to choose the `metadata.name` of the parent so that it is more intuitive for them to refer to. + ### ApplySet Member Objects #### Labels -Objects that are part of an ApplySet MUST carry two standardized labels: +Objects that are part of an ApplySet MUST carry the following label: ```yaml applyset.k8s.io/part-of: # REQUIRED -applyset.k8s.io/controller-ref: # REQUIRED ``` -The `applyset.k8s.io/part-of` is the source of truth for membership in a set. Its `` value can be chosen essentially arbitrarily (subject to the limits of label values), but MUST match the value of `applyset.k8s.io/id` on the parent (see below). As such, sourcing this value from the end user is recommended. +This `applyset.k8s.io/part-of` label is the source of truth for membership in a set. Its `` value MUST match the value of `applyset.k8s.io/id` on the parent (see [ApplySet Parent Objects](#labels-and-annotations)) and comply with the naming constraints outlined in [ApplySet Naming](#applyset-naming). -The `applyset.k8s.io/controller-ref` is used to detect collisions between applysets with the same name, which is a bad practice that cannot be reliably prevented. Its `` value will be in a machine-readible format dictated by this specification, likely either the parent's UUID or a base64 encoding of its GKNN (to be finalized during initial prototyping). Tooling that manages objects MUST set this label, and MUST verify that the identifier on the member object matches the parent before taking any action (e.g. deletion) on that member. ### ApplySet Parent Objects -ApplySet parent objects can (in theory) be of any type, though specific tools may limit the number of types they support for performance or UX reasons. While a purpose-made cluster-scoped CRD would be a logical choice, the specification has no opinion on this, so as to accommodate the many and various choices existing tools have already made. In choosing types to support as parents, tools should consider what permissions their target users typically have on their clusters; for instance, they may not have permissions to install CRDs. +Although ApplySet parent objects can (in theory) be of any type, for both performance and simplicity, we choose to limit supported types to the following: +- ConfigMap +- Secret +- custom resources, where the CRD registering them is labeled with `applyset.k8s.io/role/parent` (The value is currently ignored, but implementors should set an empty value to be forwards-compatible with future evolution of this convention.) + +This list may be extended in the future as use cases are presented. Keeping it minimal has significant benefits for the performance of any commands that list the parent objects themselves. Specific tools may further limit the types they support, but any tooling designed to identify all ApplySets must consider the exact list set out above. + +While a purpose-made cluster-scoped CRD is a logical choice, the specification has no opinion on this, so as to accommodate the many and various choices existing tools have already made. In choosing types to support as parents, tools should consider what permissions their target users typically have on their clusters; for instance, they may not have permissions to install CRDs. #### Labels and annotations @@ -466,7 +482,7 @@ ApplySets MUST also have a “parent object” in the cluster. The ApplySet obje applyset.k8s.io/id: # REQUIRED ``` -The `applyset.k8s.io/id` label is what makes the object an ApplySet parent object. As mentioned above, its `` value is replicated on set members' `applyset.k8s.io/part-of` labels, and its value can be chosen arbitrarily (subject to the limits of label values). As this value is likely to be provided by end users to identify the set they intend to operate on, allowing them to choose a value meaningful to them is recommended. +The `applyset.k8s.io/id` label is what makes the object an ApplySet parent object. Its value MUST match the value of `applyset.k8s.io/part-of` on the member objects (see [ApplySet Member Objects](#labels)), and MUST comply with the naming constraints outlined in [ApplySet Naming](#applyset-naming). Additionally, ApplySet parents MUST be labelled with: @@ -484,33 +500,31 @@ ApplySets MAY have an annotation extending their scope, which MUST be respected applyset.k8s.io/additional-namespaces: [,] # OPTIONAL ``` -The `applyset.k8s.io/additional-namespaces` annotation extends the scope of the ApplySet. By default, the scope of an ApplySet is its parent object's namespace. When the parent is cluster-scoped but refers to namespace kinds, or when the set spans multiple namespaces (which is not recommended, but allowed), this annotation an be used to extend the ApplySet's scoped to the listed namespaces. As cross-namespace ApplySets are not particularly encouraged, we do not currently optimize this further. In particular, we do not specify the GKs per namespace. We can add more annotations in future should the need arise. +The `applyset.k8s.io/additional-namespaces` annotation extends the scope of the ApplySet. By default, the scope of an ApplySet is its parent object's namespace. When the parent is cluster-scoped but refers to namespace kinds (see below), or when the set spans multiple namespaces (which is not recommended, but allowed), this annotation can be used to extend the ApplySet's scoped to the listed namespaces. As cross-namespace ApplySets are not particularly encouraged, we do not currently optimize this further. In particular, we do not specify the GKs per namespace. We can add more annotations in future should the need arise. -The value of this annotation is a comma-separated list of the names of the namespaces (other than the ApplySet namespace) in which objects are found, for example `default,kube-system,ns1,ns2`. We reserve the empty value. As with `applyset.k8s.io/contains-group-kinds`, this list of namespaces must be sorted alphabetically, and should be minimal (other than transiently during applyset mutations). +The value of this annotation is a comma-separated list of the names of the namespaces (other than the ApplySet namespace) in which objects are found, for example `default,kube-system,ns1,ns2`. We reserve the empty value. As with `applyset.k8s.io/contains-group-kinds`, this list of namespaces must be sorted alphabetically, and should be minimal (other than transiently during ApplySet mutations). #### Optional "hint" annotations The ApplySet parent object MAY also have one or more of the following annotations that help tooling identify ApplySet members more efficiently. These are annotations instead of labels because annotations can be larger, and because we definitely do not need to select on these values. -While the use of either of these annotations assists arbitrary tools in listing the applyset members, their use is not required by the specification, and tooling should not populate such annotations unless it believes itself to be the manager of an applyset. +While the use of either of these annotations assists arbitrary tools in listing the ApplySet members, their use is not required by the specification, and tooling should not populate such annotations unless it believes itself to be the manager of an ApplySet. Most tools will likely want to have some GK/object identification mechanism along these lines for performance and permissions reasons, but “porcelain” tooling can continue to do using its existing mechanisms, be they more efficient or more powerful or just easier to continue to support. -When a tool that wants to list the members of an applyset and cannot determine the list of GKs (i.e. because neither hint annotation is used), it may support “discovery”, likely warning that a full cluster crawl is being attempted. Insufficient permissions errors are likely with such functionality, and when they are encountered, the tool should warn that the membership list may be incomplete. +We may revisit this and converge on a single, mandatory hint annotation before this KEP enters beta. If feedback shows a need for it, we may also consider additional "hint" mechanisms, such as supporting a field selector on the CRD that identifies a strongly-typed list of member objects. -If feedback shows a need for it, we may in future define additional "hint" mechanisms, such as supporting a field selector on the CRD that identifies a strongly-typed list of member objects. +When a tool that wants to list the members of an ApplySet and cannot determine the list of GKs (i.e. because neither hint annotation is used), it may support “discovery”, likely warning that a full cluster crawl is being attempted. Insufficient permissions errors are likely with such functionality, and when they are encountered, the tool should warn that the membership list may be incomplete. ```yaml applyset.k8s.io/contains-group-kinds: .[,]` # OPTIONAL ``` -The `applyset.k8s.io/contains-group-kinds` annotation is an optional "hint" annotation tools can populate and use to optimize listing of member objects. Tooling not using this annotation may safely ignore it. Since the annotation on the member objects themselves remains the source of truth for set membership, tools making use of this optimization should consider also providing or periodiacally automating a resync of the hint annotation. - -When present, the value of this annotation shall be a comma separated list of the group-kinds, in the fully-qualified name format, i.e. `.`. An example annotation value might therefore look like: `certificates.cert-manager.io,configmaps,deployments.apps,secrets,services`. Note that we do not include the version; formatting is a different concern from “applyset membership”. +The `applyset.k8s.io/contains-group-kinds` annotation is an optional "hint" annotation tools can populate and use to optimize listing of member objects. Tooling not using this annotation may safely ignore it. Since the annotation on the member objects themselves remains the source of truth for set membership, tools making use of this optimization should consider also providing or periodically automating a resync of the hint annotation. -To avoid spurious updates and conflicts, the list must be sorted alphabetically. The list may include GKs where there are no resources actually labeled with the applyset-id, but to avoid churn this should be avoided and ideally only be a transitional step during applyset mutations. +When present, the value of this annotation shall be a comma separated list of the group-kinds, in the fully-qualified name format, i.e. `.`. An example annotation value might therefore look like: `certificates.cert-manager.io,configmaps,deployments.apps,secrets,services`. Note that we do not include the version; formatting is a different concern from “ApplySet membership”. -If the list in this annotation includes namespaced-scoped GKs on a cluster-scoped parent with no `applyset.k8s.io/additional-namespaces` annotation, the tooling should output an error. +To avoid spurious updates and conflicts, the list must be sorted alphabetically. The list may include GKs where there are no resources actually labeled with the ApplySet-id, but to avoid churn this should be avoided and ideally only be a transitional step during ApplySet mutations. ```yaml applyset.k8s.io/inventory: ./.[,] # OPTIONAL @@ -530,34 +544,28 @@ How the ApplySet object is specified is a tooling decision. Gitops based toolin ### ApplySet scopes +Tooling MUST have some way to optimize the queries it makes to identify member objects within scope, and it MAY opt into using one of the [standard hint annotations](#optional-hint-annotations) for that purpose, for better interoperability. We may revisit this and converge on a single, mandatory hint annotation during alpha. + Although the best practice is generally to constrain ApplySets to a single scope where possible, sometimes multi-scoped sets are unavoidable in the real world. Therefore, the mechanisms we have defined here allow for ApplySets that are cluster-scoped, multi-namespace or mixed-scoped (for example ApplySets that include installation of CRDs such as cert-manager). If the parent object is namespaced, member objects may be in that same namespace or at the cluster scope. The `applyset.k8s.io/additional-namespaces` annotation can be used to allow members in additional namespaces. This is purely additive; it is not possible to create a namespaced parent object that excludes its own namespace. If the parent object is cluster-scoped, member objects by default are at the cluster scope. The `applyset.k8s.io/additional-namespaces` annotation can be used to allow member objects in one or more namespaces. -``` -<<[UNRESOLVED @justinsb ]>> - -It is not possible to prevent an ApplySet from referring to cluster-scoped resources. Should this instead be explicitly opt-in, like cross-namespace is? That could encourage best practices and improve performance by reducing the default scope. - -<<[/UNRESOLVED]>> -``` - ### Tooling Interoperability There is a rich ecosystem of existing tooling that we hope will adopt these labels and annotations. So that different tooling can interoperate smoothly, we define some requirements for safe interoperability here. -For read operations, we expect that using different tooling shall generally be safe. As these labels do not collide with existing tooling, we would expect that objects installed with existing tooling would be invisible to the porcelain tooling until they had been updated to include the labels. We do not propose to implement “bridges” to existing tooling, rather as the proposal here is lightweight and small, it makes more sense to update the existing tooling. We may add warnings such as “applysets using an old version of X detected, upgrade to v123 of X to work with those applysets”. +For read operations, we expect that using different tooling shall generally be safe. As these labels do not collide with existing tooling, we would expect that objects installed with existing tooling would be invisible to the porcelain tooling until they had been updated to include the labels. We do not propose to implement “bridges” to existing tooling, rather as the proposal here is lightweight and small, it makes more sense to update the existing tooling. We may add warnings such as “ApplySets using an old version of X detected, upgrade to v123 of X to work with those ApplySets”. -For write operations, we need to be more careful. Deleting an applyset using the “wrong tool” should be safe, but we will likely include a confirmation if deleting an applyset using the “wrong tool”, particularly unknown tools. We expect that porcelain tools may define richer behavior on delete, so this is the equivalent of pulling the power cable on an applyset instead of performing a clean shutdown. +For write operations, we need to be more careful. Deleting an ApplySet using the “wrong tool” should be safe, but we will likely include a confirmation if deleting an ApplySet using the “wrong tool”, particularly unknown tools. We expect that porcelain tools may define richer behavior on delete, so this is the equivalent of pulling the power cable on an ApplySet instead of performing a clean shutdown. -We do not believe that update operations are safe if using the “wrong tool”, because that tooling may have additional metadata that would then not be updated. Tooling should generally reject applying on top of unknown applysets. Porcelain tooling may choose to recognize other tooling and implement specific logic there; in particular this may be useful for moving between different major versions of the same tooling. +We do not believe that update operations are safe if using the “wrong tool”, because that tooling may have additional metadata that would then not be updated. Tooling should generally reject applying on top of unknown ApplySets. Porcelain tooling may choose to recognize other tooling and implement specific logic there; in particular this may be useful for moving between different major versions of the same tooling. In order to identify usage of the "wrong tool", we rely on the `applyset.k8s.io/tooling` annotation, -which tooling can set to protect their applysets. +which tooling can set to protect their ApplySets. Specification-compliant porcelain tooling MUST recognize that -a different tool is managing the applyset and provide an appropriate error or warning. +a different tool is managing the ApplySet and provide an appropriate error or warning. We intend to explore the trade-off between safety and user-friendly behaviour here, during evolution of the feature in alpha and beyond. @@ -582,7 +590,7 @@ versions to annotation values by including a prefix like `v2:` (and we would lik `v2:[...` or `v2:{...`). Colons are not valid in namespaces nor in group-kinds, so there is no conflict with the existing (v1) usage described here. Labels cannot include a `:` character, so if we needed to version a label we can use `v2.`, -however our usage of labels is primarily around matching opaque applyset-id +however our usage of labels is primarily around matching opaque ApplySet-id tokens and thus seems unlikely to need versioning. ## Design Details: Kubectl Pruning @@ -591,18 +599,14 @@ This KEP describes both a lightweight specification and a way to use that specif ### Supported ApplySet Parent Kinds -While the ApplySet specification itself does not restrict the kinds that can be used as parent objects, existing tools typically allow a small set of options in practice. For kubectl, we propose initially supporting Secret, ConfigMap, and specially configured custom resources. - -For custom resources, we require that CRDs that define types that can be used as ApplySet objects be labeled with a label with a name of `applyset.k8s.io/role/applyset`. The value is currently ignored, but implementors should set an empty value to be forwards-compatible with future evolution of this convention. - -This proposed restriction on supported Kinds is both for simplicity, and in anticipation of a performance optimization for the `kubectl apply list-apply-sets` command that we are considering for the beta timeline. Namely, this restriction significantly reduces the number of API resources the implementation of that command would need to call. A `kubectl apply list-applysets -n ns` command would therefore do the following queries: +Since kubectl operates on the plumbing-layer concept directly, it will support the exact list of types set out in [ApplySet Parent Objects](#applyset-parent-objects). A `kubectl apply list-applysets -n ns` command would therefore do the following queries: ```bash kubectl get secret -n ns -l applyset.k8s.io/id # --only-partial-object-metadata kubectl get configmap -n ns -l applyset.k8s.io/id # --only-partial-object-metadata -for crd in $(kubectl get crd -l applyset.k8s.io/role/applyset); do -kubectl get $crd -n ns -l applyset.k8s.io/id # --only-partial-object-metadata +for cr in $(kubectl get crd -l applyset.k8s.io/role/parent); do +kubectl get $cr -n ns -l applyset.k8s.io/id # --only-partial-object-metadata done ``` @@ -610,19 +614,15 @@ Optimizations are possible here. For example we can likely cache the list of CRD In future, we may define additional “index” mechanisms here to further optimize this (controllers or webhooks that watch these labels and populate an annotation on the namespace, or support in kube-apiserver for cross-object querying). However the belief is that this is likely not needed at the current time. -A drawback of this approach is that a `list-apply-sets` command operates directly on the plumbing layer, and appears to be listing _all_ ApplySets from any tool, not just those kubectl created for pruning purposes. It is possible this will be misleading, as other tools may have used kinds beyond this restriction (they could do so even if the specification advised against it, which it currently does not). - -We may relax the restriction on supported Kinds in the future based on user feedback or a decision not to implement the `list-apply-sets` command (or a decision not to optimize it for performance over inclusiveness). - ### Efficient Listing of ApplySet Contents -We want to support efficient listing of the objects that belong to a particular applyset. In theory, this again requires the all-GK listing (with a label filter). An advantage of this approach is that this remains an option: as we implement optimizations we may also periodically run a “garbage collector” to verify that our optimizations have not leaked objects, perhaps `kubectl apply fsck` or a plugin. +We want to support efficient listing of the objects that belong to a particular ApplySet. In theory, this again requires the all-GK listing (with a label filter). An advantage of this approach is that this remains an option: as we implement optimizations we may also periodically run a “garbage collector” to verify that our optimizations have not leaked objects, perhaps `kubectl apply verify-applyset` or a plugin. -We already know the label selector for a given applyset, by convention: we take the id from the value of the `applyset.k8s.io/id` label, and that becomes the required value of the `applyset.k8s.io/part-of` label. +We already know the label selector for a given ApplySet, by convention: we take the id from the value of the `applyset.k8s.io/id` label, and that becomes the required value of the `applyset.k8s.io/part-of` label. -In order to narrow the list of GKs, kubectl will use the optional `applyset.k8s.io/contains-group-kinds` annotation described in the [optional parent object annotations](#optional-annotations) section to store the list of GKs in use. Whether those kinds are cluster-scoped or namespace-scoped are found using the normal API discovery mechanisms. +In order to narrow the list of GKs, kubectl will use the optional `applyset.k8s.io/contains-group-kinds` annotation described in the [optional parent object annotations](#optional-hint-annotations) section to store the list of GKs in use. Whether those kinds are cluster-scoped or namespace-scoped are found using the normal API discovery mechanisms. -In pseudo-code, to discover the existing members of an applyset: +In pseudo-code, to discover the existing members of an ApplySet: ```bash for-each gk in $(split group-kind-annotation); do @@ -632,26 +632,28 @@ done If the `applyset.k8s.io/additional-namespaces` annotation is present, any namespaced queries will need to be repeated for each target namespace. +If the list in the `contains-group-kinds` annotation includes namespaced-scoped GKs on a cluster-scoped parent with no `applyset.k8s.io/additional-namespaces` annotation, kubectl will output an error. + If the contains-group-kinds annotation is missing, kubectl will initially consider this an error. Based on feedback, we can consider either falling back on a (very slow) full-GK scan to populate the annotation (after confirming kubectl owns the parent), or pointing users to a separate command (similar in spirit to `fsck`) that will do so. We will add warnings/suggestions to the main "apply" flow when we detect problems that might require a full-scan / discovery. We may extend this based on user-feedback from the alpha. Based on performance feedback, we can also consider switching to the alternative `applyset.k8s.io/inventory` hint annotation. Even if we do not trust the GKNN list for deletion purposes (we cannot, as it is not the source of truth), it could be used to optimize certain specific cases, most notably the no-op scenario where the current set exactly matches the list. ### Kubectl Commands and Flags -The intention of the proposed changes is to provide a supportable replacement for the current alpha `kubectl apply --prune` semantics. Our intention is not to change the behavior of the existing `--prune` functionality, but rather to produce an alternative that users will happily and safely move to. We can likely trigger the V2-semantics when the user specifies an applyset flag, so that this is intuitive and does not break existing prune users. The proposal may evolve at the coding/PR stage, but the current plan is as follows. +The intention of the proposed changes is to provide a supportable replacement for the current alpha `kubectl apply --prune` semantics. Our intention is not to change the behavior of the existing `--prune` functionality, but rather to produce an alternative that users will happily and safely move to. We can likely trigger the V2-semantics when the user specifies an ApplySet flag, so that this is intuitive and does not break existing prune users. The proposal may evolve at the coding/PR stage, but the current plan is as follows. Required for an MVP release: - `KUBECTL_APPLYSET_ALPHA=1` environment variable: Required to expose the new flags/commands during alpha. -- `kubectl apply --prune --apply-set=[resource.version.group/]name`: The `--apply-set` flag MUST be used with `--prune` and MUST have a non-empty value when used. Its GVR component is defaulted to `secrets` when missing. This flag CANNOT be used with `-l/--selector` or with `--prune-allow-list`, and this will be validated at flag parsing time. -- `kubectl apply --prune --apply-set= --dry-run` -- `kubectl diff --prune --apply-set=` +- `kubectl apply --prune --applyset=[resource.version.group/]name`: The `--applyset` flag MUST be used with `--prune` and MUST have a non-empty value when used. Its GVR component is defaulted to `secrets` when missing. This flag CANNOT be used with `-l/--selector` or with `--prune-allow-list`, and this will be validated at flag parsing time. +- `kubectl apply --prune --applyset= --dry-run` +- `kubectl diff --prune --applyset=` Tentatively proposed for future iterations (more specific design details to follow after MVP): -- `kubectl apply generate-apply-set --selector=[key=val] --legacy-allow-list=[]`: command to migrate from the legacy pruning system to this new one. -- `kubectl apply verify-apply-set [--fix]`: `fsck`-style functionality to update the annotations on the parent applyset objects. -- `kubectl apply view-apply-set -o name|json|yaml`: A command for viewing applyset membership, ideally in a way that can be programmatically chained. -- `kubectl apply disband-apply-set `: removes the `applyset.k8s.io/id` from all members and then deletes the parent applyset object. -- `kubectl apply list-apply-sets`: view apply sets, including those managed by other tools. +- `kubectl apply generate-applyset --selector=[key=val] --legacy-allow-list=[]`: command to migrate from the legacy pruning system to this new one. +- `kubectl apply verify-applyset [] [--fix]`: `fsck`-style functionality to update the annotations on the parent ApplySet objects. +- `kubectl apply view-applyset -o name|json|yaml`: A command for viewing ApplySet membership, ideally in a way that can be programmatically chained. +- `kubectl apply disband-applyset `: removes the `applyset.k8s.io/id` from all members and then deletes the parent ApplySet object. +- `kubectl apply list-applysets`: view apply sets, including those managed by other tools. Examples: @@ -677,36 +679,36 @@ kubectl diff -n foo --prune --applyset=set1 -f . # Optional commands follow: # Extension command to verify correspondence of annotations -kubectl apply verify-apply-set myset -n foo +kubectl apply verify-applyset configmap/myset -n foo -# Extension command to verify correspondence of annotations (scoped to a namespace) -kubectl apply verify-apply-set verticalpodautoscalers.autoscaling.k8s.io/tracker -n foo +# Extension command to verify all ApplySets in the cluster +kubectl apply verify-applyset # Extension command to fix correspondence of annotations -kubectl apply verify-apply-set myset -n foo --fix +kubectl apply verify-applyset myset -n foo --fix # Extension command to list objects in namespace -kubectl apply list-objects -n ns1 –applyset=set1 +kubectl apply view-applyset myset -n ns1 ``` We intend to treat the flag and any subcommands as alpha commands initially. During alpha, users will need to set an environment variable (e.g. KUBECTL_APPLYSET_ALPHA) to make the flag available. -Commands will verify that the value of `applyset.k8s.io/tooling` has the `kubectl/` prefix before making any mutation, failing with an error if the annotation is present with any other value. It will set this label to `kubectl/vX.XX` (e.g. kubectl/v1.27) when creating/adopting resources as parent objects and update the semver as needed. At least initially, a missing tooling label or blank label value will also be considered an error, though this is not strictly required by the proposed spec and could be relaxed in the future. We may implement a `--force` flag, but this would likely be logically equivalent in outcome to a full applyset deletion and recreation, though with the potential (but not the guarantee) to be less disruptive. +Commands will verify that the value of `applyset.k8s.io/tooling` has the `kubectl/` prefix before making any mutation, failing with an error if the annotation is present with any other value. It will set this label to `kubectl/vX.XX` (e.g. kubectl/v1.27) when creating/adopting resources as parent objects and update the semver as needed. At least initially, a missing tooling label or blank label value will also be considered an error, though this is not strictly required by the proposed spec and could be relaxed in the future. We may implement a `--force` flag, but this would likely be logically equivalent in outcome to a full ApplySet deletion and recreation, though with the potential (but not the guarantee) to be less disruptive. -When `--apply-set=` is used (with no GVR), kubectl will automatically default the GVR to "secret", and will use server-side apply to create or update a Secret by that name in the targeted namespace, with the labels/annotations described here. If no namespace is specified, this is an error. Secret creation will happen at the beginning of the pruning phase rather than during the main apply operation. Server-side apply (SSA) will be used to create the Secret even if the main operation used client-side apply, and conflict forcing will be disabled regardless of its status on the main operation. Taking over an existing Secret is allowed, as long as it does not have any conflicting fields (no special criteria vs subsequent operations). +When `--applyset=` is used (with no GVR), kubectl will automatically default the GVR to "secret", and will use server-side apply to create or update a Secret by that name in the targeted namespace, with the labels/annotations described here. If no namespace is specified, this is an error. Secret creation will happen at the beginning of the pruning phase rather than during the main apply operation. Server-side apply (SSA) will be used to create the Secret even if the main operation used client-side apply, and conflict forcing will be disabled regardless of its status on the main operation. Taking over an existing Secret is allowed, as long as it does not have any conflicting fields (no special criteria vs subsequent operations). -Since there is no obvious choice for a cluster-scoped built-in resource that could be similarly chosen as the default applyset kind, we will allow the kind to optionally be specified in the `--apply-set` flag itself: `--applyset=mykind.v1.mygroup/name`. This is the same format used by `kubectl get`. When a GVR is specified in this manner, kubectl will look up the referenced object and attempt to use it as the parent (using SSA as described above for the Secret case). The referenced object MUST already exist on the cluster by the time the pruning phase begins (it may be created by the main apply operation), as it is not possible for kubectl to sanely construct arbitrary object types from scratch. +Since there is no obvious choice for a cluster-scoped built-in resource that could be similarly chosen as the default ApplySet kind, we will allow the kind to optionally be specified in the `--applyset` flag itself: `--applyset=mykind.v1.mygroup/name`. This is the same format used by `kubectl get`. When a GVR is specified in this manner, kubectl will look up the referenced object and attempt to use it as the parent (using SSA as described above for the Secret case). The referenced object MUST already exist on the cluster by the time the pruning phase begins (it may be created by the main apply operation), as it is not possible for kubectl to sanely construct arbitrary object types from scratch. -In future, we may support a applyset object being provided as part of the input resources. For example, if the input resources contain an object with the `applyset.k8s.io/id=` label, this could be interpreted as the parent, and the `--apply-set` flag could be made optional. However, this adds complexity and has potential downsides and edge cases to handle (e.g. multiple labelled objects), so we will do so in response to user feedback, if at all. +In future, we may support a ApplySet object being provided as part of the input resources. For example, if the input resources contain an object with the `applyset.k8s.io/id=` label, this could be interpreted as the parent, and the `--applyset` flag could be made optional. However, this adds complexity and has potential downsides and edge cases to handle (e.g. multiple labelled objects), so we will do so in response to user feedback, if at all. -When pruning with `--apply-set`, kubectl will delete objects that are labeled as part of the applyset of objects, but are not in the list of objects being applied. We expect to reuse the existing prune logic and behavior here, except that we will select objects differently (although as existing prune is also based on label selection, we may be able to reuse the bulk of the label-selection logic also). Dry-run will be supported, as will `kubectl diff --prune --apply-set=id`. +When pruning with `--applyset`, kubectl will delete objects that are labeled as part of the ApplySet of objects, but are not in the list of objects being applied. We expect to reuse the existing prune logic and behavior here, except that we will select objects differently (although as existing prune is also based on label selection, we may be able to reuse the bulk of the label-selection logic also). Dry-run will be supported, as will `kubectl diff --prune --applyset=id`. -The `--prune` flag will continue to be required for all pruning operations to ensure communication of intent for this destructive feature. The `--apply-set` flag has no meaning on its own and specifying it without `--prune` is an error. We will not support any of the scoping flags used by the previous pruning feature, that is, `--prune-allowlist`, `-l/--selector` and `--all`. These are considered conflicting pruning "modes", and specifying them alongside `--apply-set` will fail flag validation. Our goal is to support the existing safe workflows, not the full permutations of all flags. The allowlist function in particular should be redundant with our improved discovery. What meaning the label selector flag would have if allowed is unclear, and we will need to collaborate with kubectl users to understand their true intent if there is demand for compatibility with that flag. +The `--prune` flag will continue to be required for all pruning operations to ensure communication of intent for this destructive feature. The `--applyset` flag has no meaning on its own and specifying it without `--prune` is an error. We will not support any of the scoping flags used by the previous pruning feature, that is, `--prune-allowlist`, `-l/--selector` and `--all`. These are considered conflicting pruning "modes", and specifying them alongside `--applyset` will fail flag validation. Our goal is to support the existing safe workflows, not the full permutations of all flags. The allowlist function in particular should be redundant with our improved discovery. What meaning the label selector flag would have if allowed is unclear, and we will need to collaborate with kubectl users to understand their true intent if there is demand for compatibility with that flag. -The `--namespace` flag will be required when using any namespaced parent, including the default Secret. Because that flag throws a mismatch error when the set contains resources with heterogeneous namespaces, this limits the scope of applyset-based pruning in kubectl specifically beyond what the spec proposed strictly requires. Specifically, in kubectl, applysets spanning multiple namespaces MUST use a cluster-scoped parent object. We believe this limitation is reasonable and encourages best practices, but we could consider relaxing this position (e.g. using the applyset-in-input option described above) based on user feedback. When applicable, kubectl will ensure that all "visited" namespaces (as defined in the current operational code) are named by the sum of the parent's own namespace (if any) and the `applyset.k8s.io/additional-namespaces` annotation. +The `--namespace` flag will be required when using any namespaced parent, including the default Secret. Because that flag throws a mismatch error when the set contains resources with heterogeneous namespaces, this limits the scope of ApplySet-based pruning in kubectl specifically beyond what the spec proposed strictly requires. Specifically, in kubectl, ApplySets spanning multiple namespaces MUST use a cluster-scoped parent object. We believe this limitation is reasonable and encourages best practices, but we could consider relaxing this position (e.g. using the ApplySet-in-input option described above) based on user feedback. When applicable, kubectl will ensure that all "visited" namespaces (as defined in the current operational code) are named by the sum of the parent's own namespace (if any) and the `applyset.k8s.io/additional-namespaces` annotation. -We will detect “overlapping” applysets where objects already have a different applyset label, and initially treat this +We will detect “overlapping” ApplySets where objects already have a different ApplySet label, and initially treat this an error. During implementation of the alpha we will explore to what extent we can optimize this overlap discovery, particularly in conjunction with server-side-apply which does not require an object read before applying. @@ -717,7 +719,7 @@ optimize differently for kubectl. In the worst case, we will have to fetch the objects before applying (with a set of label-filtered LIST requests), we will explore to what extent that can be amortized over other kubectl operations in alpha. One interesting option may be to use the fieldManager, -choosing a fieldManager that includes the applyset ID to automatically +choosing a fieldManager that includes the ApplySet ID to automatically detect conflicts (by _not_ specifying force); we intend to explore how this looks in practice and whether other options present themselves. @@ -725,25 +727,25 @@ We differentiate between "adoption" (taking over management of a set of objects created by another tool), vs "migration" (taking over management of a set of objects created with the existing pruning mechanism). -We will not support "adoption" of existing applysets initially, other than +We will not support "adoption" of existing ApplySets initially, other than by re-applying "over the top". Based on user feedback, we may require a flag -to adopt existing objects / applysets. +to adopt existing objects / ApplySets. In the alpha scope, we will explore suitable "migration" tooling for moving from existing `--prune` objects. Note that migration is not trivial, in that different users may expect different behaviors with regard to the GKs selected or the treatment of objects having/lacking the `last-application-configuration` annotation. We intend to create an explicit migration subcommand on `apply`, e.g. -`kubectl apply generate-apply-set --selector=[key=val] --legacy-allow-list=[]`, +`kubectl apply generate-applyset --selector=[key=val] --legacy-allow-list=[]`, rather than trying to overload the "normal flow" apply command. ### Security Considerations -Generally RBAC gives us the permissions we need to operate safely here. No special permissions are granted - for example there is no “backdoor” to read objects simply because they are part of an applyset. In order to mark an object as part of an applyset, we need permission to write to that object. If we have permission to update an applyset object, we can “leak” objects from the optimized search, but we can support a “fsck” scan that does not optimize the search, and generally the ability to mutate the applyset carries this risk. Using a more privileged object, such as a secret or a dedicated CRD can limit this risk. +Generally RBAC gives us the permissions we need to operate safely here. No special permissions are granted - for example there is no “backdoor” to read objects simply because they are part of an ApplySet. In order to mark an object as part of an ApplySet, we need permission to write to that object. If we have permission to update an ApplySet object, we can “leak” objects from the optimized search, but we can support a “fsck” scan that does not optimize the search, and generally the ability to mutate the ApplySet carries this risk. Using a more privileged object, such as a secret or a dedicated CRD can limit this risk. Known Risks: -- A user without delete permission but with update permission could mark an object as part of an applyset, and then an administrator could inadvertently delete the object as part of their next apply/prune. This is also true of the current pruning implementation (by setting the last-applied-configuration annotation to any value). Mitigation: We will support the dry-run functionality for pruning. Webhooks or future enhancements to RBAC/CEL may allow for granular permission on labels. +- A user without delete permission but with update permission could mark an object as part of an ApplySet, and then an administrator could inadvertently delete the object as part of their next apply/prune. This is also true of the current pruning implementation (by setting the last-applied-configuration annotation to any value). Mitigation: We will support the dry-run functionality for pruning. Webhooks or future enhancements to RBAC/CEL may allow for granular permission on labels. ### Test Plan @@ -759,7 +761,7 @@ when drafting this test plan. [testing-guidelines]: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md --> -[ ] I/we understand the owners of the involved components may require updates to +[x] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement. @@ -830,7 +832,7 @@ label based pruning, rather than a revolutionary new approach, to try to enable At some point we might deprecate the existing `--prune` functionality, to encourage users to migrate. A suitable timeline would probably be to begin deprecation at beta, and to -not remove the functionality until at least applyset reaches GA + 1 version. However, we +not remove the functionality until at least ApplySet reaches GA + 1 version. However, we intend to gather feedback from early alphas here - in particular we want to discover: * Are there `--prune` use-cases we do not cover? @@ -1292,8 +1294,8 @@ Alternatively, we could omit the `part-of` label entirely (which leaves no means ### OwnerRefs -We could use ownerRefs to track applyset membership. A significant advantage of ownerRefs is that pruning is done automatically by the kube-apiserver, which runs a garbage collection algorithm to automatically delete resources that are no longer referenced. -However today the apiserver does not support an efficient way to query by ownerRef (unlike labels, where we can specify a label selector to the kube-apiserver). This means we can’t efficiently list the objects in an applyset, nor can we efficiently support a dry-run / preview (without listing all the objects). Moreover, there is no support for cross-namespace ownerRefs, nor for a namespace-scoped object owning a cluster-scoped object. These are not blockers per-se, in that as a community we control the full-stack. However, the scoping issues are more fundamental and have meant that existing tooling such as helm has not used ownerRefs, so this would likely be a barrier to adoption by existing tooling. We do not preclude tooling from using ownerRefs; we are simply proposing standardizing the labels to provide interoperability with existing tooling and the existing kube-apiserver. +We could use ownerRefs to track ApplySet membership. A significant advantage of ownerRefs is that pruning is done automatically by the kube-apiserver, which runs a garbage collection algorithm to automatically delete resources that are no longer referenced. +However today the apiserver does not support an efficient way to query by ownerRef (unlike labels, where we can specify a label selector to the kube-apiserver). This means we can’t efficiently list the objects in an ApplySet, nor can we efficiently support a dry-run / preview (without listing all the objects). Moreover, there is no support for cross-namespace ownerRefs, nor for a namespace-scoped object owning a cluster-scoped object. These are not blockers per-se, in that as a community we control the full-stack. However, the scoping issues are more fundamental and have meant that existing tooling such as helm has not used ownerRefs, so this would likely be a barrier to adoption by existing tooling. We do not preclude tooling from using ownerRefs; we are simply proposing standardizing the labels to provide interoperability with existing tooling and the existing kube-apiserver. ### ManagedFields From a88ff933f663b9c49fb46f7fc9bf80e662047cd3 Mon Sep 17 00:00:00 2001 From: Justin Santa Barbara Date: Thu, 26 Jan 2023 18:43:22 -0500 Subject: [PATCH 10/19] Remove leftover paragraph (#5) Not an alternative rejected any more, given applyset.k8s.io/inventory --- keps/sig-cli/3659-kubectl-apply-prune/README.md | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index 427e46eba0f..e81df0c2f74 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -1282,16 +1282,6 @@ Why should this KEP _not_ be implemented? ## Alternatives -### Full GKNN listing - -Instead of encoding a list of GKs to scope in, we could encode a the full list of GKNN object references, making the ApplySet parent object a (somewhat) human-readable inventory of the set. The reason for not choosing this approach is that we do not think it would actually allow us to further optimize the implementation in practice, and that its additional detail would make it more prone to desynchronization. - -The reason it does not optimize performance in practice is that we're considering the source of truth for membership to be the `part-of` annotations on the resources themselves. This is useful for visibility and for ownership conflict avoidance, but it means we must retrieve the objects themselves to check the source of truth rather than relying on the GVKNN. Since individual GET calls are far more expensive than LISTs in the common case for pruning, in practice, we would end up extracting the GK list from any GKNN list and make the same calls we would have with just a GK list. If it is deemed worthwhile, we could indeed do this, and it would allow an additional layer of in-band drift detection via comparison of the precise list to the set of current labelled resources. - -That said, the GKNN approach could likely be used to increase efficiency in a particularly common scenario: recognition that the set has not changed. We could choose to trust the listing in this scenario to avoid making any queries at all. A standard annotation for storing GKNN information is already part of this proposal, and we could switch the kubectl implementation to it based on experience with the alpha if desired. - -Alternatively, we could omit the `part-of` label entirely (which leaves no means of ownership conflict management), or consider the GKNN list the source of truth (which leaves a much wider vector for object leakage in practice than GK listing does, in our opinion). - ### OwnerRefs We could use ownerRefs to track ApplySet membership. A significant advantage of ownerRefs is that pruning is done automatically by the kube-apiserver, which runs a garbage collection algorithm to automatically delete resources that are no longer referenced. From 8abef138a1327d1b53960634513fdc57af4ff41d Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Thu, 26 Jan 2023 19:28:49 -0500 Subject: [PATCH 11/19] Justin has always been coauthor --- keps/sig-cli/3659-kubectl-apply-prune/README.md | 1 - keps/sig-cli/3659-kubectl-apply-prune/kep.yaml | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index e81df0c2f74..7c730f351ce 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -134,7 +134,6 @@ tags, and then generate with `hack/update-toc.sh`. - [Implementation History](#implementation-history) - [Drawbacks](#drawbacks) - [Alternatives](#alternatives) - - [Full GKNN listing](#full-gknn-listing) - [OwnerRefs](#ownerrefs) - [ManagedFields](#managedfields) - [Infrastructure Needed (Optional)](#infrastructure-needed-optional) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/kep.yaml b/keps/sig-cli/3659-kubectl-apply-prune/kep.yaml index ab516424862..e637b4f6a9c 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/kep.yaml +++ b/keps/sig-cli/3659-kubectl-apply-prune/kep.yaml @@ -2,6 +2,7 @@ title: KEP Template kep-number: 3659 authors: - "@KnVerey" + - "@justinsb" owning-sig: sig-cli participating-sigs: [] status: provisional From 95d226cd4d6ddea5ca8b235f0f293c81b734f727 Mon Sep 17 00:00:00 2001 From: Justin Santa Barbara Date: Fri, 27 Jan 2023 11:26:44 -0500 Subject: [PATCH 12/19] KEP-3659: production readiness etc (#4) Fill in the testing/ PRR sections. --- .../3659-kubectl-apply-prune/README.md | 72 +++++++++++++++++-- 1 file changed, 68 insertions(+), 4 deletions(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index 7c730f351ce..51151d262fe 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -780,6 +780,10 @@ However, if complete unit test coverage is not possible, explain the reason of i together with explanation why this is acceptable. --> +We will strive for a reasonable trade-off between agility +and code coverage, consistent with the high standards of +the kubernetes projects. + -- : +We will add e2e tests to verify the core operational flows, +in particular: + +* applying a set of objects ("set1") as an applyset and with pruning (no changes) +* applying a partially overlaping set of objects ("set2") as an applyset with dry-run pruning (no changes made but differences reported) +* applying set2 without pruning (new objects added, no pruning) +* applying set2 as an applyset with dry-run pruning (no changes made but pruning reported) +* applying set2 as an applyset with pruning (pruning operates as expected) +* applying set2 as an applyset with pruning again (no changes) ### Graduation Criteria @@ -837,6 +847,12 @@ intend to gather feedback from early alphas here - in particular we want to disc * Are there `--prune` use-cases we do not cover? * Do existing `--prune` users migrate enthusiastically (without any "nudge" from deprecation)? +#### Alpha + +- Feature implemented behind env var +- Initial e2e tests completed and enabled +- Positive user feedback gathered + +Plan for alpha is that users must explicitly opt-in with `KUBECTL_APPLYSET_ALPHA`, +existing functionality will remain. + ### Version Skew Strategy +Functionality is client-side only. The functionality does not reference +versioned fields or kinds (i.e. uses group-kinds, not group-version-kinds; +uses metadata.labels instead of dedicated fields). + ## Production Readiness Review Questionnaire +If the user opts-in, then they will get the new functionality. For alpha, +the existing functionality will remain the default for alpha. We do not +plan to replace the existing functionality, users will always need to +opt-in by specifying an applyset-id flag. + ###### Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)? +Yes, this is client-side only. + ###### What happens if we reenable the feature if it was previously rolled back? +Mixing and matching current pruning with new pruning and non-pruning +might cause objects to not be pruned, as with today when mixing pruning +modes. + ###### Are there any tests for feature enablement/disablement? +We will maintain tests for "prune v2" and "prune v1", until +such time as prune v1 (technically still in alpha) is removed. + ### Rollout, Upgrade and Rollback Planning +The number of calls should be comparable to prune-v1. We will investigate +optimizing these calls (e.g. using PartialObjectMetadata). + +We will also investigate during alpha replacing client-side throttling with +parallel behaviour that better makes matches priority and fairness. + ###### Will enabling / using this feature result in introducing new API types? +No + ###### Will enabling / using this feature result in any new calls to the cloud provider? +No + ###### Will enabling / using this feature result in increasing size or count of the existing API objects? +Small increase: +* New applyset objects +* Small "applyset" label on all child objects + ###### Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs? +Impact not expected to be measurable. + ###### Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components? +Impact expected to be negligible. + ### Troubleshooting -We will maintain tests for "prune v2" and "prune v1", until -such time as prune v1 (technically still in alpha) is removed. +We'll add unit tests (or cases) that cover ApplySet-based apply/prune and maintain it alongside existing coverage for "prune v1", until such time as prune v1 (technically still in alpha) is removed. +We'll make sure there's test coverage for any modification of or interaction with the v1 implementation to make sure it doesn't regress. We will also include coverage for the supported flag permutations when the ApplySet alpha is enabled. ### Rollout, Upgrade and Rollback Planning From 9db89439821a87673560737f4387ecee03c11ec3 Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Tue, 7 Feb 2023 20:35:38 -0500 Subject: [PATCH 18/19] ID vs name fixes --- keps/sig-cli/3659-kubectl-apply-prune/README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index d162e9ef82c..60574aa4ee8 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -100,7 +100,7 @@ tags, and then generate with `hack/update-toc.sh`. - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional) - [Risks and Mitigations](#risks-and-mitigations) - [Design Details: ApplySet Specification](#design-details-applyset-specification) - - [ApplySet Naming](#applyset-naming) + - [ApplySet Identification](#applyset-identification) - [ApplySet Member Objects](#applyset-member-objects) - [Labels](#labels) - [ApplySet Parent Objects](#applyset-parent-objects) @@ -440,22 +440,22 @@ Implicit in this proposal are a few assumptions: - An object can be part of at most one ApplySet. This is a limitation, but seems to be a good one in that objects that are part of multiple ApplySets are complicated both conceptually for users and in terms of implementation behaviour. - An ApplySet object can be part of another ApplySet (sub-ApplySets). -### ApplySet Naming +### ApplySet Identification -Each ApplySet MUST have an ID that can be used to uniquely identify the parent and member objects via the label selector conventions outlined in the following sections. As such, the name: +Each ApplySet MUST have an ID that can be used to uniquely identify the parent and member objects via the label selector conventions outlined in the following sections. As such, the ID: * is subject to the normal limits of label values -* MUST be the base64 encoding of the hash of the GKNN, in the form `base64(sha256(...))`, using the URL safe encoding of RFC4648. +* MUST be the base64 encoding of the hash of the GKNN of the parent object, in the form `base64(sha256(...))`, using the URL safe encoding of RFC4648. -The second restriction is intended to protect against "id impersonation" attacks; +The second restriction is intended to protect against "ID impersonation" attacks; we will likely evaluate specifics here during alpha (for example whether to include an empty string for a namespace on cluster-scoped objects). When operating against an existing applyset, tooling MUST verify the applyset against -the generation mechanism here. Tooling MUST return an error if the applyset id does -not match, though it MAY support some sort of force or repair operation instead, though +the generation mechanism here. Tooling MUST return an error if the applyset ID does +not match the GKNN of the parent, though it MAY support some sort of force or repair operation instead, though this should require confirmation of some kind from the user ("type yes" or a flag). -This applyset ID does not need to be used for the `metadata.name` of the parent object. +This applyset ID clearly cannot be used for the `metadata.name` of the parent object since the ID is partially composed of that field's value. Tooling should likely allow end users to choose the `metadata.name` of the parent so that it is more intuitive for them to refer to. From b347756461679f62cf985e7a6b0fd0bc28ea9fd2 Mon Sep 17 00:00:00 2001 From: Katrina Verey Date: Wed, 8 Feb 2023 16:20:31 -0500 Subject: [PATCH 19/19] Fixes from soltysh's review --- .../3659-kubectl-apply-prune/README.md | 34 ++++++++++--------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/keps/sig-cli/3659-kubectl-apply-prune/README.md b/keps/sig-cli/3659-kubectl-apply-prune/README.md index 60574aa4ee8..cbc3c42440b 100644 --- a/keps/sig-cli/3659-kubectl-apply-prune/README.md +++ b/keps/sig-cli/3659-kubectl-apply-prune/README.md @@ -18,13 +18,13 @@ To get started with this template: - [x] **Fill out as much of the kep.yaml file as you can.** At minimum, you should fill in the "Title", "Authors", "Owning-sig", "Status", and date-related fields. -- [ ] **Fill out this file as best you can.** +- [x] **Fill out this file as best you can.** At minimum, you should fill in the "Summary" and "Motivation" sections. These should be easy if you've preflighted the idea of the KEP with the appropriate SIG(s). -- [ ] **Create a PR for this KEP.** +- [x] **Create a PR for this KEP.** Assign it to people in the SIG who are sponsoring this process. -- [ ] **Merge early and iterate.** +- [x] **Merge early and iterate.** Avoid getting hung up on specific details and instead aim to get the goals of the KEP clarified and merged quickly. The best way to do this is to just start with the high-level sections and fill out details incrementally in @@ -144,18 +144,18 @@ tags, and then generate with `hack/update-toc.sh`. Items marked with (R) are required *prior to targeting to a milestone / release*. -- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) -- [ ] (R) KEP approvers have approved the KEP status as `implementable` -- [ ] (R) Design details are appropriately documented -- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) +- [x] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [x] (R) KEP approvers have approved the KEP status as `implementable` +- [x] (R) Design details are appropriately documented +- [x] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors) - [ ] e2e Tests for all Beta API Operations (endpoints) - [ ] (R) Ensure GA e2e tests meet requirements for [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) - [ ] (R) Minimum Two Week Window for GA e2e tests to prove flake free -- [ ] (R) Graduation criteria is in place +- [x] (R) Graduation criteria is in place - [ ] (R) [all GA Endpoints](https://github.com/kubernetes/community/pull/1806) must be hit by [Conformance Tests](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md) -- [ ] (R) Production readiness review completed -- [ ] (R) Production readiness review approved -- [ ] "Implementation History" section is up-to-date for milestone +- [x] (R) Production readiness review completed +- [x] (R) Production readiness review approved +- [x] "Implementation History" section is up-to-date for milestone - [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] - [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes @@ -271,7 +271,7 @@ Options: ``` -The reason for this stagnation is that the implementation has fundamental limitations that limit performance and cause unexpected behaviours. +The reason for this stagnation is that the implementation has fundamental limitations that affect performance and cause unexpected behaviours. Acknowledging that pruning could not be progressed out of alpha in its current form, SIG CLI created a proof of concept for an alternative implmentation in the [cli-utils](https://github.com/kubernetes-sigs/cli-utils) repo in 2019 (initially [moved over](https://github.com/kubernetes-sigs/cli-utils/pull/1) from [cli-experimental#13](https://github.com/kubernetes-sigs/cli-experimental/pull/13)). This implementation was proposed in [KEP 810](https://github.com/kubernetes/enhancements/pull/810/files), which did not reach consensus and was ultimately closed. In the subsequent three years, work continued on the proof of concept, and other ecosystem tools (notably `kpt live apply`) have been using it successfully while the canonicial implementation in k/k has continued to stagnate. @@ -654,7 +654,7 @@ Based on performance feedback, we can also consider switching to the alternative The intention of the proposed changes is to provide a supportable replacement for the current alpha `kubectl apply --prune` semantics. Our intention is not to change the behavior of the existing `--prune` functionality, but rather to produce an alternative that users will happily and safely move to. We can likely trigger the V2-semantics when the user specifies an ApplySet flag, so that this is intuitive and does not break existing prune users. The proposal may evolve at the coding/PR stage, but the current plan is as follows. Required for an MVP release: -- `KUBECTL_APPLYSET_ALPHA=1` environment variable: Required to expose the new flags/commands during alpha. +- `KUBECTL_APPLYSET=1` environment variable: Required to expose the new flags/commands during alpha. - `kubectl apply --prune --applyset=[resource.version.group/]name`: The `--applyset` flag MUST be used with `--prune` and MUST have a non-empty value when used. Its GVR component is defaulted to `secrets` when missing. This flag CANNOT be used with `-l/--selector` or with `--prune-allow-list`, and this will be validated at flag parsing time. - `kubectl apply --prune --applyset= --dry-run` - `kubectl diff --prune --applyset=` @@ -703,7 +703,7 @@ kubectl apply view-applyset myset -n ns1 ``` -We intend to treat the flag and any subcommands as alpha commands initially. During alpha, users will need to set an environment variable (e.g. KUBECTL_APPLYSET_ALPHA) to make the flag available. +We intend to treat the flag and any subcommands as alpha commands initially. During alpha, users will need to set an environment variable (e.g. KUBECTL_APPLYSET) to make the flag available. Commands will verify that the value of `applyset.k8s.io/tooling` has the `kubectl/` prefix before making any mutation, failing with an error if the annotation is present with any other value. It will set this label to `kubectl/vX.XX` (e.g. kubectl/v1.27) when creating/adopting resources as parent objects and update the semver as needed. At least initially, a missing tooling label or blank label value will also be considered an error, though this is not strictly required by the proposed spec and could be relaxed in the future. We may implement a `--force` flag, but this would likely be logically equivalent in outcome to a full ApplySet deletion and recreation, though with the potential (but not the guarantee) to be less disruptive. @@ -946,7 +946,7 @@ enhancement: cluster required to make on upgrade, in order to make use of the enhancement? --> -Plan for alpha is that users must explicitly opt-in with `KUBECTL_APPLYSET_ALPHA`, +Plan for alpha is that users must explicitly opt-in with `KUBECTL_APPLYSET`, existing functionality will remain. ### Version Skew Strategy @@ -1017,7 +1017,7 @@ well as the [existing list] of feature gates. - Describe the mechanism: Functionality is client-side only. Plan for alpha is that users must explicitly - opt-in with `KUBECTL_APPLYSET_ALPHA`, existing functionality will remain. + opt-in with `KUBECTL_APPLYSET`, existing functionality will remain. - Will enabling / disabling the feature require downtime of the control plane? @@ -1354,6 +1354,8 @@ Major milestones might include: - when the KEP was retired or superseded --> +- Feb 2023: KEP accepted and alpha implementation started + ## Drawbacks