ApplySet : kubectl apply --prune redesign and graduation strategy

[KnVerey]

Enhancement Description

• One-line enhancement description (can be used as a release note): This KEP proposes a safer and more performant implementation for kubectl apply's pruning feature, which enables cleanup of previously applied objects that have been removed from desired state provided to the current apply.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-cli/3659-kubectl-apply-prune
• Discussion Link: The SIG CLI maintainer track talk at Kubecon NA 2021 included a segment on the problem. Most recently, this was discussed in a Contributor Summit unconference session at Kubecon NA 2022 as well as async in this background doc, which was also raised in this SIG CLI meeting. In these recent conversations, there has been consensus that we need to finally do something to move this perma-alpha forward for 1.27; hence the creation of this KEP.
• Primary contact (assignee): @KnVerey
• Responsible SIGs: CLI
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.27
  • Beta release target (x.y): TBD
  • Stable release target (x.y): TBD
• Alpha
  • KEP (k/enhancements) update PR(s):
    • KEP-3659: ApplySet: kubectl apply --prune redesign and graduation strategy #3661
  • Code (k/k) update PR(s):
    • Introduce CLI for ApplySet-based pruning kubernetes#115979
    • prunev2: Basic pruning logic kubernetes#116205
    • ApplySet: allow custom resources to be parent objects kubernetes#116353
    • Applyset dry run tests + ID value kubernetes#116265
    • Create and update the ApplySet parent object kubernetes#116243
    • kubectl diff: refactor tracker into a separate type kubernetes#116579
    • kubectl prunev2: simple e2e/integration test kubernetes#116599
    • kubectl prunev2: Refactor the applyset to be more reusable kubernetes#116580
    • kubectl prunev2: issue discovery requests in parallel kubernetes#116584
  • Docs (k/website) update PR(s):
    • Add docs for ApplySet-based pruning in kubectl apply website#39818

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig cli

[soltysh]

/assign @KnVerey
/stage alpha
/milestone v1.27
/label lead-opted-in

[marosset]

Hello @KnVerey 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for v1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: v1.27
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

For this enhancement, it looks like #3661 will address most of these requirements.
Please be sure to also:

• Update the status to implementable in the kep.yaml
• Record the PRR approval in a new file under https://github.com/kubernetes/enhancements/tree/master/keps/prod-readiness/sig-cli

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well.
Thank you!

[KnVerey]

@marosset I believe all of the requirements have been met with the merging of #3661 today!

[marosset]

This enhancement meets now meets all of the requirements to be tracked in v1.27.
Thanks!

[KnVerey]

Docs placeholder PR: kubernetes/website#39818

[marosset]

Hi @KnVerey 👋,

Checking in as we approach 1.27 code freeze at 17:00 PDT on Tuesday 14th March 2023.

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are fully merged by the code freeze deadline.

Please let me know if there are any PRs in k/k I should be tracking for this KEP.

As always, we are here to help should questions come up. Thanks!

[LukeMwila]

Hi @KnVerey, I’m reaching out from the 1.27 Release Docs team. This enhancement is marked as ‘Needs Docs’ for the 1.27 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.27 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by March 16. For more information, please take a look at Documenting for a release to familiarize yourself with the documentation requirements for the release.

Please feel free to reach out with any questions. Thanks!

[marosset]

Unfortunately the implementation PRs associated with this enhancement have not merged by code-freeze so this enhancement is getting removed from the release.

If you would like to file an exception please see https://github.com/kubernetes/sig-release/blob/master/releases/EXCEPTIONS.md

/milestone clear
/remove-label tracked/yes
/label tracked/no

[KnVerey]

Hi @marosset they did make the release actually! You can see them here: https://github.com/orgs/kubernetes/projects/128/views/2. I will update the issue description. The only feature we were originally targeting as part of the first alpha that did not make it was kubectl diff support. The featureset in kubectl apply is complete as intended for this release.

[marosset]

/milestonve v1.27
/label tracked/yes
/remove-label tracked/no

[marosset]

Hi @marosset they did make the release actually! You can see them here: https://github.com/orgs/kubernetes/projects/128/views/2. I will update the issue description. The only feature we were originally targeting as part of the first alpha that did not make it was kubectl diff support. The featureset in kubectl apply is complete as intended for this release.

@KnVerey I added this issue back into v1.27.
Thanks for linking all the PRs above!

[sftim]

BTW, nearly all the labels we register are using subdomains of kubernetes.io. This KEP is using *.k8s.io keys.

If you want to make life easier for end users, get an exception in to change the labels, before beta (ideally, before the alpha release). I know it's a bit later, but it looks like we missed that detail in earlier reviews.

See https://kubernetes.io/docs/reference/labels-annotations-taints/ for the list of registered keys that we use for labels and annotations.

[KnVerey]

/milestone v1.27

(there was a typo in the last attempt to apply this)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[armingerten]

The only feature we were originally targeting as part of the first alpha that did not make it was kubectl diff support. The featureset in kubectl apply is complete as intended for this release.

Is there already a timeline when ApplySets will be supported by kubectl diff? There also seems to be another stale issue about this: kubernetes/kubectl#1435

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[btrepp]

Is this really not planned now?. That's kinda disappointing, it was a really good feature and was looking forward to it.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[btrepp]

Could the bot be disabled?.
This is probably a long term issue and the bot closing it, is just making noise in the issue.

If applysets aren't going forward that's okay and the issue should be closed, but the bot is adding no value on this thread

[sftim]

/reopen

/lifecycle frozen
My opinion: when we have in-tree code (alpha or later), we should freeze the KEP issue until that in-tree code is either stable or removed.

[k8s-ci-robot]

@sftim: Reopened this issue.

In response to this:

/reopen

/lifecycle frozen
My opinion: when we have in-tree code (alpha or later), we should freeze the KEP issue until that in-tree code is either stable or removed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sftim]

Oh, hang on. This never made it into the tree and nobody is driving it forward?
/remove-lifeycle frozen
/close not-planned

[k8s-ci-robot]

@sftim: Closing this issue, marking it as "Not Planned".

In response to this:

Oh, hang on. This never made it into the tree and nobody is driving it forward.
/remove-lifeycle frozen
/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[btrepp]

Oh, hang on. This never made it into the tree and nobody is driving it forward.
/remove-lifeycle frozen
/close not-planned

Thanks for looking into it.

I know this command is available in the cli in an alpha state, and is on the k8s docs

https://kubernetes.io/docs/tasks/manage-kubernetes-objects/declarative-config/#kubectl-apply-prune-1

Should it maybe be removed, or the docs updated if it isn't going to be developed?

It's a feature that solves some problems with managing what is being deployed, and was pretty usable in alpha. I know I adopted it in place of heavy operators like flux and argocd, so I imagine people will find the docs and use it, even though it is alpha.

[uhthomas]

Please don't remove applysets 😭

[schlichtanders]

I am also happily using applysets. The applyset alpha version so far is super stable and works way better then the previous --prune support. I am also using it as a really great lightweight alternative to flux/argocd.

[sftim]

Right, it is in tree. If we can't maintain it we'll have to find a way to make things work though; unmaintained code is a risk all in itself.

/lifecycle frozen
/reopen

[k8s-ci-robot]

@sftim: Reopened this issue.

In response to this:

Right, it is in tree. If we can't maintain it we'll have to find a way to make things work though; unmaintained code is a risk all in itself.

/lifecycle frozen
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[Barsonax]

Would love to see apply sets getting some more love