From 16290dc2193aec80de77b9cb226df516dd75ed30 Mon Sep 17 00:00:00 2001 From: Engin Diri Date: Fri, 4 Oct 2024 17:39:28 +0200 Subject: [PATCH] feat: add External Secrets Operator (ESO) integration page (#12978) * feat: add External Secrets Operator (ESO) integration page * Update content/docs/esc/integrations/kubernetes/external-secrets-operator.md Co-authored-by: James Denyer --------- Co-authored-by: James Denyer --- content/docs/esc/integrations/_index.md | 3 +- .../esc/integrations/kubernetes/_index.md | 22 + .../kubernetes/external-secrets-operator.md | 476 ++++++++++++++++++ .../{ => kubernetes}/kubernetes.md | 4 +- 4 files changed, 502 insertions(+), 3 deletions(-) create mode 100644 content/docs/esc/integrations/kubernetes/_index.md create mode 100644 content/docs/esc/integrations/kubernetes/external-secrets-operator.md rename content/docs/esc/integrations/{ => kubernetes}/kubernetes.md (98%) diff --git a/content/docs/esc/integrations/_index.md b/content/docs/esc/integrations/_index.md index 7e54092c4069..5e055b692947 100644 --- a/content/docs/esc/integrations/_index.md +++ b/content/docs/esc/integrations/_index.md @@ -34,7 +34,8 @@ ESC also integrates with tools like Direnv, Terraform, and Docker to help manage ## Kubernetes -- [Kubernetes](/docs/esc/integrations/kubernetes) +- [Kubernetes](/docs/esc/integrations/kubernetes/kubernetes) +- [External Secrets Operator (ESO)](/docs/esc/integrations/kubernetes/external-secrets-operator) ## Developer tools diff --git a/content/docs/esc/integrations/kubernetes/_index.md b/content/docs/esc/integrations/kubernetes/_index.md new file mode 100644 index 000000000000..6a6278e0b540 --- /dev/null +++ b/content/docs/esc/integrations/kubernetes/_index.md @@ -0,0 +1,22 @@ +--- +title: Kubernetes +title_tag: Kubernetes integrations | Pulumi ESC +h1: ESC Kubernetes integrations +meta_desc: Pulumi ESC integrates with Kubernetes to manage configurations, credentials, and kubeconfig files. +menu: + esc: + identifier: esc-kubernetes-integrations + parent: esc-integrations + weight: 5 +aliases: + - /docs/esc/kubernetes-integrations +--- + +Pulumi ESC's rich metadata and support for popular configuration formats enables easy integration with Kubernetes. This allows you to manage configurations, credentials, and `kubeconfig` files for Kubernetes clusters, and to interact with Kubernetes tools such as `kubectl` and `helm`. Additionally, Pulumi ESC integrates with different tools in the Kubernetes ecosystem, such as the Pulumi Kubernetes provider and the External Secrets Operator (ESO). + +To learn how to configure Kubernetes with Pulumi ESC, see the following topics: + +| Tool | Description | +|------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Kubernetes](/docs/esc/integrations/kubernetes/kubernetes) | Pulumi ESC integrates with Kubernetes to manage configurations, credentials, and kubeconfig files, with kubectl and helm, and Pulumi Kubernetes provider. | +| [External Secrets Operator (ESO)](/docs/esc/integrations/kubernetes/external-secrets-operator) | Pulumi ESC integrates with the External Secrets Operator (ESO) to manage and deliver secrets in Kubernetes clusters. | | diff --git a/content/docs/esc/integrations/kubernetes/external-secrets-operator.md b/content/docs/esc/integrations/kubernetes/external-secrets-operator.md new file mode 100644 index 000000000000..a192aafdba39 --- /dev/null +++ b/content/docs/esc/integrations/kubernetes/external-secrets-operator.md @@ -0,0 +1,476 @@ +--- +title: External Secrets Operator (ESO) +title_tag: Integrate with External Secrets Operator (ESO) | Pulumi ESC +h1: "Pulumi ESC: Integrate with External Secrets Operator (ESO)" +meta_desc: Pulumi ESC integrates with the External Secrets Operator (ESO) to manage and deliver secrets in Kubernetes clusters. +weight: 2 +menu: + esc: + identifier: esc-external-secrets-operator + parent: esc-kubernetes-integrations +aliases: +- /docs/esc/other-integrations/external-secrets-operator/ +--- + +## Overview + +[External Secrets Operator](https://external-secrets.io/latest/) is a Kubernetes operator that integrates external secret management systems with Kubernetes. By using External Secrets Operator, you have several advantages over Kubernetes native secrets: + +- Sensitive data is stored and managed by an external service outside the Kubernetes cluster and this leads to better security and compliance. +- External Secrets Operator supports multiple different sources, so you can use the same operator to manage secrets and configuration from different sources. +- Take advantage of the advanced features of the secret provider, such as encryption of data at rest and scenarios like secret rotation. + +Since version `0.10.0` External Secrets Operator supports Pulumi ESC as a secret provider. + +## Prerequisites + +In this guide, you will learn how to deploy the External Secrets Operator and use it to manage secrets stored in Pulumi ESC. Before you start, make sure you have the following prerequisites: + +- the [Pulumi ESC CLI](/docs/esc-cli/) +- a Kubernetes cluster +- [kubectl](https://kubernetes.io/releases/download/#kubectl) +- [helm](https://helm.sh/docs/intro/install/) + +## Deploy External Secrets Operator + +### Deploy using Helm + +#### Install from Helm Chart Repository + +```bash +helm repo add external-secrets https://charts.external-secrets.io +helm repo update + +helm upgrade --install external-secrets external-secrets/external-secrets \ + --namespace external-secrets \ + --create-namespace \ + --wait +``` + +#### Create secret containing Pulumi access token + +```bash +kubectl create secret generic pulumi-access-token -from-literal=PULUMI_ACCESS_TOKEN=${PULUMI_ACCESS_TOKEN} \ + --namespace external-secrets +``` + +#### Create ClusterSecretStore + +Now you can create a [ClusterSecretStore](https://external-secrets.io/main/api/clustersecretstore/) resource that will tell External Secrets Operator to use Pulumi ESC as a secret provider. + +If you want to limit the access by namespace, you can create a [SecretStore](https://external-secrets.io/main/api/secretstore/) resource instead, which is scoped to a single namespace. + +```yaml +cat < --name external-secrets-operator +``` + +Add the following code to the `index.ts`, `index.py`, or `main.go` file: + +{{< chooser language "typescript,python,go" />}} + +{{% choosable language typescript %}} + +```typescript +import * as pulumi from "@pulumi/pulumi"; +import * as kubernetes from "@pulumi/kubernetes"; + +const config = new pulumi.Config(); + +const externalSecretsNamespace = new kubernetes.core.v1.Namespace("external-secrets-namespace", { + metadata: { + name: "external-secrets", + } +}); +const externalSecretsRelease = new kubernetes.helm.v3.Release("external-secrets-release", { + chart: "external-secrets", + version: "0.10.4", + namespace: externalSecretsNamespace.metadata.apply(metadata => metadata.name), + repositoryOpts: { + repo: "https://charts.external-secrets.io", + }, +}); +const patSecret = new kubernetes.core.v1.Secret("patSecret", { + metadata: { + namespace: externalSecretsNamespace.metadata.apply(metadata => metadata.name), + name: "pulumi-access-token", + }, + stringData: { + PULUMI_ACCESS_TOKEN: config.require("pulumi-pat"), + }, + type: "Opaque", +}); + +const clusterSecretStore = new kubernetes.apiextensions.CustomResource("cluster-secret-store", { + apiVersion: "external-secrets.io/v1beta1", + kind: "ClusterSecretStore", + metadata: { + name: "secret-store", + }, + spec: { + provider: { + pulumi: { + organization: pulumi.getOrganization(), + project: "dirien", + environment: "hello-world", + accessToken: { + secretRef: { + name: patSecret.metadata.name, + key: "PULUMI_ACCESS_TOKEN", + namespace: patSecret.metadata.namespace, + }, + }, + }, + }, + }, +}, { dependsOn: externalSecretsRelease }); + +const externalSecret = new kubernetes.apiextensions.CustomResource("external-secret", { + apiVersion: "external-secrets.io/v1beta1", + kind: "ExternalSecret", + metadata: { + name: "secret", + namespace: "default", + }, + spec: { + data: [ + { + secretKey: "esc-secret", + remoteRef: { + key: "hello", + } + } + ], + refreshInterval: "20s", + secretStoreRef: { + kind: clusterSecretStore.kind, + name: clusterSecretStore.metadata.name, + } + }, +}, { dependsOn: externalSecretsRelease }); +``` + +{{% /choosable %}} + +{{% choosable language python %}} + +```python +import pulumi +import pulumi_kubernetes as kubernetes + +config = pulumi.Config() + +external_secrets_namespace = kubernetes.core.v1.Namespace( + "external-secrets-namespace", + metadata=kubernetes.meta.v1.ObjectMetaArgs(name="external-secrets"), +) + +external_secrets_release = kubernetes.helm.v3.Release( + "external-secrets-release", + chart="external-secrets", + version="0.10.4", + namespace=external_secrets_namespace.metadata.apply(lambda metadata: metadata.name), + repository_opts=kubernetes.helm.v3.RepositoryOptsArgs( + repo="https://charts.external-secrets.io" + ), +) + +pat_secret = kubernetes.core.v1.Secret( + "patSecret", + metadata=kubernetes.meta.v1.ObjectMetaArgs( + namespace=external_secrets_namespace.metadata.apply( + lambda metadata: metadata.name + ), + name="pulumi-access-token", + ), + string_data={"PULUMI_ACCESS_TOKEN": config.require("pulumi-pat")}, + type="Opaque", +) + +cluster_secret_store = kubernetes.apiextensions.CustomResource( + "cluster-secret-store", + api_version="external-secrets.io/v1beta1", + kind="ClusterSecretStore", + metadata=kubernetes.meta.v1.ObjectMetaArgs(name="secret-store"), + spec={ + "provider": { + "pulumi": { + "organization": pulumi.get_organization(), + "project": "dirien", + "environment": "hello-world", + "accessToken": { + "secretRef": { + "name": pat_secret.metadata.name, + "key": "PULUMI_ACCESS_TOKEN", + "namespace": pat_secret.metadata.namespace, + } + }, + } + } + }, + opts=pulumi.ResourceOptions(depends_on=[external_secrets_release]), +) + +external_secret = kubernetes.apiextensions.CustomResource( + "external-secret", + api_version="external-secrets.io/v1beta1", + kind="ExternalSecret", + metadata=kubernetes.meta.v1.ObjectMetaArgs(name="secret", namespace="default"), + spec={ + "data": [{"secretKey": "esc-secret", "remoteRef": {"key": "hello"}}], + "refreshInterval": "20s", + "secretStoreRef": { + "kind": cluster_secret_store.kind, + "name": cluster_secret_store.metadata.name, + }, + }, + opts=pulumi.ResourceOptions(depends_on=[external_secrets_release]), +) +``` + +{{% /choosable %}} + +{{% choosable language go %}} + +```go +package main + +import ( + "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes" + "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions" + "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1" + "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3" + metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1" + "github.com/pulumi/pulumi/sdk/v3/go/pulumi" + "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config" +) + +func main() { + pulumi.Run(func(ctx *pulumi.Context) error { + conf := config.New(ctx, "") + + externalSecretsNamespace, err := v1.NewNamespace(ctx, "external-secrets-namespace", &v1.NamespaceArgs{ + Metadata: &metav1.ObjectMetaArgs{ + Name: pulumi.String("external-secrets"), + }, + }) + if err != nil { + return err + } + + externalSecretsRelease, err := helm.NewRelease(ctx, "external-secrets-release", &helm.ReleaseArgs{ + Chart: pulumi.String("external-secrets"), + Version: pulumi.String("0.10.4"), + Namespace: externalSecretsNamespace.Metadata.Name(), + RepositoryOpts: &helm.RepositoryOptsArgs{ + Repo: pulumi.String("https://charts.external-secrets.io"), + }, + }) + if err != nil { + return err + } + + patSecret, err := v1.NewSecret(ctx, "patSecret", &v1.SecretArgs{ + Metadata: &metav1.ObjectMetaArgs{ + Namespace: externalSecretsNamespace.Metadata.Name(), + Name: pulumi.String("pulumi-access-token"), + }, + StringData: pulumi.StringMap{ + "PULUMI_ACCESS_TOKEN": conf.RequireSecret("pulumi-pat"), + }, + Type: pulumi.String("Opaque"), + }) + if err != nil { + return err + } + + clusterSecretStore, err := apiextensions.NewCustomResource(ctx, "cluster-secret-store", &apiextensions.CustomResourceArgs{ + ApiVersion: pulumi.String("external-secrets.io/v1beta1"), + Kind: pulumi.String("ClusterSecretStore"), + Metadata: &metav1.ObjectMetaArgs{ + Name: pulumi.String("secret-store"), + }, + OtherFields: kubernetes.UntypedArgs{ + "spec": pulumi.Map{ + "provider": pulumi.Map{ + "pulumi": pulumi.Map{ + "organization": pulumi.String(ctx.Organization()), + "project": pulumi.String("dirien"), + "environment": pulumi.String("hello-world"), + "accessToken": pulumi.Map{ + "secretRef": pulumi.Map{ + "name": patSecret.Metadata.Name(), + "key": pulumi.String("PULUMI_ACCESS_TOKEN"), + "namespace": patSecret.Metadata.Namespace(), + }, + }, + }, + }, + }}, + }, pulumi.DependsOn([]pulumi.Resource{externalSecretsRelease})) + if err != nil { + return err + } + + _, err = apiextensions.NewCustomResource(ctx, "external-secret", &apiextensions.CustomResourceArgs{ + ApiVersion: pulumi.String("external-secrets.io/v1beta1"), + Kind: pulumi.String("ExternalSecret"), + Metadata: &metav1.ObjectMetaArgs{ + Name: pulumi.String("secret"), + Namespace: pulumi.String("default"), + }, + OtherFields: kubernetes.UntypedArgs{ + "spec": pulumi.Map{ + "data": pulumi.Array{ + pulumi.Map{ + "secretKey": pulumi.String("esc-secret"), + "remoteRef": pulumi.Map{ + "key": pulumi.String("hello"), + }, + }, + }, + "refreshInterval": pulumi.String("20s"), + "secretStoreRef": pulumi.Map{ + "kind": clusterSecretStore.Kind, + "name": clusterSecretStore.Metadata.Name(), + }, + }, + }, + }, pulumi.DependsOn([]pulumi.Resource{externalSecretsRelease})) + if err != nil { + return err + } + + return nil + }) +} +``` + +{{% /choosable %}} + +You can then deploy the Pulumi program using below command as you would normally do with any Pulumi program: + +```bash +pulumi up +``` + +#### Cleanup + +To remove the External Secrets Operator and the created resources, you can use the following commands: + +```bash +pulumi destroy +``` + +## Push Secrets to Pulumi ESC + +The Pulumi ESC provider for External Secrets Operator supports also [PushSecrets](https://external-secrets.io/latest/api/pushsecret/). This feature allows you to push secrets from a Kubernetes cluster to Pulumi ESC. This is useful, if you have for example other operators that create secrets in the cluster and you want automatically push them to Pulumi ESC for further management. + +Here is an example of a [PushSecret](https://external-secrets.io/latest/api/pushsecret/) resource: + +```yaml +apiVersion: external-secrets.io/v1alpha1 +kind: PushSecret +metadata: + name: push-secret-example +spec: + refreshInterval: 10s + selector: + secret: + name: + secretStoreRefs: + - kind: ClusterSecretStore + name: secret-store + data: + - match: + secretKey: + remoteRef: + remoteKey: +``` + +## Conclusion + +This tutorial showed you how to deploy the External Secrets Operator and use it to manage secrets stored in Pulumi ESC. This gives you the ability to elevate Pulumi ESC as the single source of truth for your secrets even when you are not using Pulumi to manage your Kubernetes resources. + +As we continue to improve the integration between Pulumi ESC and External Secrets Operator, we highly recommend you to check the [External Secrets Operator Pulumi ESC](https://external-secrets.io/latest/provider/pulumi/) for the latest features. diff --git a/content/docs/esc/integrations/kubernetes.md b/content/docs/esc/integrations/kubernetes/kubernetes.md similarity index 98% rename from content/docs/esc/integrations/kubernetes.md rename to content/docs/esc/integrations/kubernetes/kubernetes.md index bc453da398fb..e022a20dfea4 100644 --- a/content/docs/esc/integrations/kubernetes.md +++ b/content/docs/esc/integrations/kubernetes/kubernetes.md @@ -1,13 +1,13 @@ --- title: Kubernetes title_tag: Integrate with Kubernetes | Pulumi ESC -h1: Kubernetes +h1: "Pulumi ESC: Integrate with Kubernetes" meta_desc: Pulumi ESC integrates with Kubernetes to manage configurations, credentials, and kubeconfig files, with kubectl and helm, and Pulumi Kubernetes provider. weight: 2 menu: esc: identifier: esc-kubernetes - parent: esc-integrations + parent: esc-kubernetes-integrations aliases: - /docs/esc/other-integrations/kubernetes/ ---