Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): implement platform destroy command #2261

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

rohantmp
Copy link
Contributor

@rohantmp rohantmp commented Nov 4, 2024

What issue type does this pull request address? (keep at least one, remove the others)
/kind feature

What does this pull request do? Which issues does it resolve? (use resolves #<issue_number> if possible)
resolves ENG-3980

Please provide a short message that should be published in the vcluster release notes
Add a new cli command: vcluster platform destroy

What else do we need to know?

The command will wait for each resource type to finish cleaning up before proceeding to the next, so that resource types that depend on each other will not prevent cleanup. (Remove a cluster object before a virtualclusterinstance running on it, etc)

vcluster platform destroy (no args) on v4.0.0:

16:28:54 info found platform installation in namespace "vcluster-platform"
16:28:54 debug checking all "virtualclusterinstances"
16:28:54 info deleting virtualclusterinstances p-default/v1
16:28:56 debug checking all "virtualclusterinstances"
16:28:58 debug checking all "virtualclusterinstances"
16:29:00 debug checking all "virtualclusterinstances"
16:29:02 debug checking all "virtualclusterinstances"
16:29:04 debug checking all "virtualclusterinstances"
16:29:06 debug checking all "virtualclusterinstances"
16:29:08 debug checking all "virtualclusterinstances"
16:29:10 debug checking all "virtualclusterinstances"
16:29:12 debug checking all "virtualclusterinstances"
16:29:14 debug checking all "virtualclusterinstances"
16:29:14 debug legacy resource "virtualclusters" not found in discovery, skipping
16:29:14 debug checking all "devpodworkspaceinstances"
16:29:14 debug checking all "spaceinstances"
16:29:14 debug checking all "virtualclustertemplates"
16:29:14 info deleting virtualclustertemplates vcluster-pro-isolated-template
16:29:14 info deleting virtualclustertemplates vcluster-pro-template
16:29:16 debug checking all "virtualclustertemplates"
16:29:16 debug checking all "devpodenvironmenttemplates"
16:29:16 debug checking all "devpodworkspacetemplates"
16:29:16 info deleting devpodworkspacetemplates aws
16:29:16 info deleting devpodworkspacetemplates azure
16:29:16 info deleting devpodworkspacetemplates digitalocean
16:29:17 info deleting devpodworkspacetemplates ecs
16:29:17 info deleting devpodworkspacetemplates gcloud
16:29:17 info deleting devpodworkspacetemplates kubernetes
16:29:18 debug checking all "devpodworkspacetemplates"
16:29:18 debug checking all "clusterroletemplates"
16:29:18 info deleting clusterroletemplates loft-apps-admin
16:29:18 info deleting clusterroletemplates loft-apps-creator
16:29:19 info deleting clusterroletemplates loft-apps-viewer
16:29:19 info deleting clusterroletemplates loft-cluster-admin
16:29:19 info deleting clusterroletemplates loft-cluster-authenticated
16:29:19 info deleting clusterroletemplates loft-cluster-extra-space-admin-rules
16:29:19 info deleting clusterroletemplates loft-cluster-extra-space-viewer-rules
16:29:19 info deleting clusterroletemplates loft-cluster-space-admin
16:29:19 info deleting clusterroletemplates loft-cluster-space-viewer
16:29:19 info deleting clusterroletemplates loft-clusters-admin
16:29:19 info deleting clusterroletemplates loft-clusters-creator
16:29:19 info deleting clusterroletemplates loft-impersonator
16:29:19 info deleting clusterroletemplates loft-management-activity-viewer
16:29:19 info deleting clusterroletemplates loft-management-admin
16:29:20 info deleting clusterroletemplates loft-management-authenticated
16:29:20 info deleting clusterroletemplates loft-management-project-admin
16:29:20 info deleting clusterroletemplates loft-management-project-user
16:29:20 info deleting clusterroletemplates loft-management-project-viewer
16:29:20 info deleting clusterroletemplates loft-secrets-admin
16:29:21 info deleting clusterroletemplates loft-secrets-creator
16:29:21 info deleting clusterroletemplates loft-secrets-viewer
16:29:21 info deleting clusterroletemplates loft-teams-admin
16:29:21 info deleting clusterroletemplates loft-teams-creator
16:29:21 info deleting clusterroletemplates loft-teams-viewer
16:29:22 info deleting clusterroletemplates loft-templates-admin
16:29:22 info deleting clusterroletemplates loft-templates-creator
16:29:22 info deleting clusterroletemplates loft-templates-viewer
16:29:22 info deleting clusterroletemplates loft-users-admin
16:29:22 info deleting clusterroletemplates loft-users-creator
16:29:23 info deleting clusterroletemplates loft-users-viewer
16:29:23 debug checking all "clusterroletemplates"
16:29:23 debug checking all "spacetemplates"
16:29:23 info deleting spacetemplates isolated-space
16:29:25 debug checking all "spacetemplates"
16:29:25 debug checking all "apps"
16:29:25 info deleting apps argocd
16:29:25 info deleting apps central-hostpath-mapper
16:29:25 info deleting apps cert-manager
16:29:25 info deleting apps example-bash
16:29:25 info deleting apps example-helm
16:29:25 info deleting apps example-manifests
16:29:25 info deleting apps ingress-nginx
16:29:25 info deleting apps jspolicy
16:29:25 info deleting apps kube-prometheus-stack
16:29:26 info deleting apps opentelemetry-collector
16:29:27 debug checking all "apps"
16:29:27 debug legacy resource "spaceconstraints" not found in discovery, skipping
16:29:27 debug checking all "tasks"
16:29:27 debug checking all "clusterquotas"
16:29:27 debug checking all "projects"
16:29:27 info deleting projects default
16:29:29 debug checking all "projects"
16:29:31 debug checking all "projects"
16:29:33 debug checking all "projects"
16:29:33 debug checking all "runners"
16:29:33 info deleting runners local
16:29:35 debug checking all "runners"
16:29:35 debug checking all "clusters"
16:29:35 info deleting clusters loft-cluster
16:29:37 debug checking all "clusters"
16:29:37 debug checking all "clusteraccesses"
16:29:37 info deleting clusteraccesses loft-admins
16:29:39 debug checking all "clusteraccesses"
16:29:39 debug checking all "networkpeers"
16:29:39 info deleting networkpeers loft-7c7685654c-9sg5x.control-plane
16:29:41 debug checking all "networkpeers"
16:29:41 info deleting networkpeers loft-7c7685654c-9sg5x.local.runner
16:29:43 debug checking all "networkpeers"
16:29:43 debug checking all "teams"
16:29:43 info deleting teams loft-admins
16:29:45 debug checking all "teams"
16:29:47 debug checking all "teams"
16:29:49 debug checking all "teams"
16:29:51 debug checking all "teams"
16:29:53 debug checking all "teams"
16:29:55 debug checking all "teams"
16:29:57 debug checking all "teams"
16:29:59 debug checking all "teams"
16:29:59 debug checking all "users"
16:29:59 info deleting users admin
16:30:01 debug checking all "users"
16:30:01 debug checking all "sharedsecrets"
16:30:01 debug checking all "accesskeys"
16:30:01 info deleting accesskeys loft-control-plane
16:30:01 info deleting accesskeys loft-runner-local-bzbjm
16:30:03 debug checking all "accesskeys"
16:30:03 debug legacy resource "localclusteraccesses" not found in discovery, skipping
16:30:03 debug legacy resource "localteams" not found in discovery, skipping
16:30:03 debug legacy resource "localusers" not found in discovery, skipping
16:30:03 info Uninstalling vCluster Platform...
16:30:03 info Executing command: helm uninstall loft --kube-context kind-v1 --namespace vcluster-platform

16:30:03 done Successfully uninstalled vCluster Platform

16:30:03 info deleting clusterrole "loft-agent-cluster"
16:30:03 info deleting clusterrole "loft-runnner-cluster"
16:30:03 info deleting clusterrole "loft-vcluster-cluster"
16:30:03 info deleting clusterrolebinding "loft-agent-cluster-binding"
16:30:03 info deleting clusterrolebinding "loft-runnner-cluster-binding"
16:30:03 info deleting clusterrolebinding "loft-vcluster-cluster-binding"
16:30:03 info deleting customresourcedefinition accesskeys.storage.loft.sh
16:30:03 info deleting customresourcedefinition apps.storage.loft.sh
16:30:03 info deleting customresourcedefinition clusteraccesses.storage.loft.sh
16:30:03 info deleting customresourcedefinition clusterquotas.storage.loft.sh
16:30:04 info deleting customresourcedefinition clusterroletemplates.storage.loft.sh
16:30:04 info deleting customresourcedefinition clusters.storage.loft.sh
16:30:04 info deleting customresourcedefinition devpodenvironmenttemplates.storage.loft.sh
16:30:04 info deleting customresourcedefinition devpodworkspaceinstances.storage.loft.sh
16:30:04 info deleting customresourcedefinition devpodworkspacetemplates.storage.loft.sh
16:30:04 info deleting customresourcedefinition networkpeers.storage.loft.sh
16:30:04 info deleting customresourcedefinition projects.storage.loft.sh
16:30:04 info deleting customresourcedefinition runners.storage.loft.sh
16:30:04 info deleting customresourcedefinition sharedsecrets.storage.loft.sh
16:30:04 info deleting customresourcedefinition spaceinstances.storage.loft.sh
16:30:04 info deleting customresourcedefinition spacetemplates.storage.loft.sh
16:30:05 info deleting customresourcedefinition tasks.storage.loft.sh
16:30:05 info deleting customresourcedefinition teams.storage.loft.sh
16:30:05 info deleting customresourcedefinition users.storage.loft.sh
16:30:05 info deleting customresourcedefinition virtualclusterinstances.storage.loft.sh
16:30:05 info deleting customresourcedefinition virtualclustertemplates.storage.loft.sh
16:30:06 info deleting namespace "vcluster-platform"

vcluster platform destroy (no args) on v3.4.9:

16:27:18 debug checking all "virtualclusterinstances"
16:27:18 debug checking all "virtualclusters"
16:27:18 debug checking all "devpodworkspaceinstances"
16:27:18 debug checking all "spaceinstances"
16:27:18 debug checking all "virtualclustertemplates"
16:27:18 info deleting virtualclustertemplates vcluster-pro-isolated-template
16:27:18 info deleting virtualclustertemplates vcluster-pro-template
16:27:20 debug checking all "virtualclustertemplates"
16:27:20 info resource "devpodenvironmenttemplates" not found in discovery, skipping
16:27:20 debug checking all "devpodworkspacetemplates"
16:27:20 info deleting devpodworkspacetemplates aws
16:27:20 info deleting devpodworkspacetemplates azure
16:27:20 info deleting devpodworkspacetemplates digitalocean
16:27:20 info deleting devpodworkspacetemplates ecs
16:27:20 info deleting devpodworkspacetemplates gcloud
16:27:20 info deleting devpodworkspacetemplates kubernetes
16:27:22 debug checking all "devpodworkspacetemplates"
16:27:22 debug checking all "clusterroletemplates"
16:27:22 info deleting clusterroletemplates loft-apps-admin
16:27:22 info deleting clusterroletemplates loft-apps-creator
16:27:22 info deleting clusterroletemplates loft-apps-viewer
16:27:22 info deleting clusterroletemplates loft-cluster-admin
16:27:22 info deleting clusterroletemplates loft-cluster-authenticated
16:27:22 info deleting clusterroletemplates loft-cluster-extra-space-admin-rules
16:27:22 info deleting clusterroletemplates loft-cluster-extra-space-viewer-rules
16:27:22 info deleting clusterroletemplates loft-cluster-space-admin
16:27:22 info deleting clusterroletemplates loft-cluster-space-viewer
16:27:23 info deleting clusterroletemplates loft-clusters-admin
16:27:23 info deleting clusterroletemplates loft-clusters-creator
16:27:23 info deleting clusterroletemplates loft-impersonator
16:27:23 info deleting clusterroletemplates loft-management-activity-viewer
16:27:23 info deleting clusterroletemplates loft-management-admin
16:27:24 info deleting clusterroletemplates loft-management-authenticated
16:27:24 info deleting clusterroletemplates loft-management-project-admin
16:27:24 info deleting clusterroletemplates loft-management-project-user
16:27:24 info deleting clusterroletemplates loft-management-project-viewer
16:27:24 info deleting clusterroletemplates loft-secrets-admin
16:27:25 info deleting clusterroletemplates loft-secrets-creator
16:27:25 info deleting clusterroletemplates loft-secrets-viewer
16:27:25 info deleting clusterroletemplates loft-teams-admin
16:27:25 info deleting clusterroletemplates loft-teams-creator
16:27:25 info deleting clusterroletemplates loft-teams-viewer
16:27:26 info deleting clusterroletemplates loft-templates-admin
16:27:26 info deleting clusterroletemplates loft-templates-creator
16:27:26 info deleting clusterroletemplates loft-templates-viewer
16:27:26 info deleting clusterroletemplates loft-users-admin
16:27:26 info deleting clusterroletemplates loft-users-creator
16:27:27 info deleting clusterroletemplates loft-users-viewer
16:27:27 debug checking all "clusterroletemplates"
16:27:27 debug checking all "spacetemplates"
16:27:27 info deleting spacetemplates isolated-space
16:27:29 debug checking all "spacetemplates"
16:27:29 debug checking all "apps"
16:27:29 info deleting apps argocd
16:27:29 info deleting apps central-hostpath-mapper
16:27:29 info deleting apps cert-manager
16:27:29 info deleting apps example-bash
16:27:29 info deleting apps example-helm
16:27:29 info deleting apps example-manifests
16:27:29 info deleting apps ingress-nginx
16:27:29 info deleting apps jspolicy
16:27:29 info deleting apps kube-prometheus-stack
16:27:30 info deleting apps opentelemetry-collector
16:27:31 debug checking all "apps"
16:27:31 debug checking all "spaceconstraints"
16:27:31 debug checking all "tasks"
16:27:31 debug checking all "clusterquotas"
16:27:31 debug checking all "projects"
16:27:31 info deleting projects default
16:27:33 debug checking all "projects"
16:27:35 debug checking all "projects"
16:27:37 debug checking all "projects"
16:27:37 debug checking all "runners"
16:27:37 info deleting runners local
16:27:39 debug checking all "runners"
16:27:39 debug checking all "clusters"
16:27:39 info deleting clusters loft-cluster
16:27:41 debug checking all "clusters"
16:27:41 debug checking all "clusteraccesses"
16:27:41 info deleting clusteraccesses loft-admins
16:27:43 debug checking all "clusteraccesses"
16:27:43 debug checking all "networkpeers"
16:27:43 info deleting networkpeers loft-b67bf7475-r7knc.control-plane
16:27:45 debug checking all "networkpeers"
16:27:45 debug checking all "teams"
16:27:45 info deleting teams loft-admins
16:27:47 debug checking all "teams"
16:27:49 debug checking all "teams"
16:27:51 debug checking all "teams"
16:27:53 debug checking all "teams"
16:27:55 debug checking all "teams"
16:27:57 debug checking all "teams"
16:27:59 debug checking all "teams"
16:27:59 debug checking all "users"
16:27:59 info deleting users admin
16:28:01 debug checking all "users"
16:28:01 debug checking all "sharedsecrets"
16:28:01 debug checking all "accesskeys"
16:28:01 info deleting accesskeys loft-agent-loft-cluster-47287
16:28:01 info deleting accesskeys loft-control-plane
16:28:01 info deleting accesskeys loft-runner-local-xsblh
16:28:03 debug checking all "accesskeys"
16:28:03 debug checking all "localclusteraccesses"
16:28:03 debug checking all "localteams"
16:28:03 debug checking all "localusers"
16:28:03 info Uninstalling vCluster Platform...
16:28:03 info Executing command: helm uninstall loft --kube-context kind-v3 --namespace vcluster-platform

16:28:03 done Successfully uninstalled vCluster Platform

16:28:03 info deleting clusterrole "loft-agent-cluster"
16:28:03 info deleting clusterrole "loft-runnner-cluster"
16:28:03 info deleting clusterrole "loft-vcluster-cluster"
16:28:03 info deleting clusterrolebinding "loft-agent-cluster-binding"
16:28:03 info deleting clusterrolebinding "loft-runnner-cluster-binding"
16:28:03 info deleting clusterrolebinding "loft-vcluster-cluster-binding"
16:28:03 info deleting customresourcedefinition accesskeys.storage.loft.sh
16:28:03 info deleting customresourcedefinition apps.storage.loft.sh
16:28:03 info deleting customresourcedefinition clusteraccesses.storage.loft.sh
16:28:03 info deleting customresourcedefinition clusterquotas.storage.loft.sh
16:28:03 info deleting customresourcedefinition clusterroletemplates.storage.loft.sh
16:28:03 info deleting customresourcedefinition clusters.storage.loft.sh
16:28:03 info deleting customresourcedefinition devpodworkspaceinstances.storage.loft.sh
16:28:03 info deleting customresourcedefinition devpodworkspacetemplates.storage.loft.sh
16:28:04 info deleting customresourcedefinition localclusteraccesses.storage.loft.sh
16:28:04 info deleting customresourcedefinition localteams.storage.loft.sh
16:28:04 info deleting customresourcedefinition localusers.storage.loft.sh
16:28:04 info deleting customresourcedefinition networkpeers.storage.loft.sh
16:28:04 info deleting customresourcedefinition projects.storage.loft.sh
16:28:04 info deleting customresourcedefinition runners.storage.loft.sh
16:28:04 info deleting customresourcedefinition sharedsecrets.storage.loft.sh
16:28:05 info deleting customresourcedefinition spaceconstraints.storage.loft.sh
16:28:05 info deleting customresourcedefinition spaceinstances.storage.loft.sh
16:28:05 info deleting customresourcedefinition spacetemplates.storage.loft.sh
16:28:05 info deleting customresourcedefinition tasks.storage.loft.sh
16:28:05 info deleting customresourcedefinition teams.storage.loft.sh
16:28:06 info deleting customresourcedefinition users.storage.loft.sh
16:28:06 info deleting customresourcedefinition virtualclusterinstances.storage.loft.sh
16:28:06 info deleting customresourcedefinition virtualclusters.storage.loft.sh
16:28:06 info deleting customresourcedefinition virtualclustertemplates.storage.loft.sh
16:28:07 info deleting namespace "vcluster-platform"

Copy link

netlify bot commented Nov 4, 2024

Deploy Preview for vcluster-docs canceled.

Built without sensitive environment variables

Name Link
🔨 Latest commit 4b4293b
🔍 Latest deploy log https://app.netlify.com/sites/vcluster-docs/deploys/6728a68da0578900084bad6e

@rohantmp rohantmp force-pushed the destroyCommand branch 5 times, most recently from 474e568 to fa9d5e1 Compare November 4, 2024 10:31
@rohantmp rohantmp marked this pull request as ready for review November 4, 2024 11:00
Copy link
Member

@ThomasK33 ThomasK33 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we putting the delete command into the start folder?
That seems a bit counter-intuitive 😅

@@ -141,7 +161,7 @@ func (l *LoftStarter) prepare() error {
contextToLoad = l.Context
} else if platformConfig.LastInstallContext != "" && platformConfig.LastInstallContext != contextToLoad {
contextToLoad, err = l.Log.Question(&survey.QuestionOptions{
Question: product.Replace("Seems like you try to use 'loft start' with a different kubernetes context than before. Please choose which kubernetes context you want to use"),
Question: product.Replace(fmt.Sprintf("Seems like you try to use 'loft %s' with a different kubernetes context than before. Please choose which kubernetes context you want to use", l.CommandName)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also rename the command from 'loft %s' to 'vcluster platform %s' here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will fix!

},
}

startCmd := &cobra.Command{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess you meant destroyCmd 😉


startCmd.Flags().StringVar(&cmd.Context, "context", "", "The kube context to use for installation")
startCmd.Flags().StringVar(&cmd.Namespace, "namespace", "", "The namespace vCluster platform is installed in")
startCmd.Flags().BoolVar(&cmd.DeleteNamespace, "delete-namespace", true, "Whether to delete the namespace or not")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to ignore: Might be handy to have something like --ignore-not-found, which does not error when there's no platfrom.

@@ -0,0 +1,248 @@
package start
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this should be in a destroy package!?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, makes sense! I can put the client initializers and validation in the clihelper

)

// define the order of resource deletion
var resourceOrder = []string{
Copy link
Contributor

@johannesfrey johannesfrey Nov 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just stating the obvious, but this means we always have to keep the resources in sync manually here because we have to respect the ordering!?
Do you know what would happen if we would just delete all resources coming from discoveryClient.ServerResourcesForGroupVersion("storage.loft.sh/v1") with a Background deletion policy without a specific order?!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think we need to do some e2e where we run vclusterctl commands against a platform instance.

I think the deletion would hang in most cases if we did an unordered delete because many things like accesskeys don't have finalizers

@rohantmp
Copy link
Contributor Author

rohantmp commented Nov 5, 2024

I'll move it into a destroy folder and move some of the common options out into clihelper

@deniseschannon deniseschannon added this to the v0.22 milestone Nov 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants