Skip to content

Commit

Permalink
Add backup tool (#315)
Browse files Browse the repository at this point in the history
* Enhance the `restore` tool

Signed-off-by: loheagn <[email protected]>

* Add `backup` tool

Signed-off-by: loheagn <[email protected]>
  • Loading branch information
loheagn authored May 27, 2022
1 parent 2117b20 commit ae8296e
Show file tree
Hide file tree
Showing 7 changed files with 483 additions and 73 deletions.
51 changes: 37 additions & 14 deletions hack/tool/backup_restore/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,20 @@ This go module is a command line tool to back up and restore the configuration a

It has two subcommands `backup` and `restore`.

## `backup`

`backup` can be used to back up the Configuration object managed by terraform-controller and the Terraform state (if the configuration uses the Terraform Kubernetes backend).

The main usage of the `backup` subcommand is:

```shell
go main.go backup --name <name of the Configuration> --namespace <namespace of the Configuration>
```

Then you will get the `cofiguration.yaml` and the `state.json` in the workdir.

Next, you can restore the Configuration and the Terraform state in another Kubernetes cluster using the `restore` subcommand.

## `restore`

The main usage of the `restore` subcommand is to import an "outside" Terraform instance (maybe created by the terraform command line or managed by another terraform-controller before) to the terraform-controller in the target Kubernetes without recreating the cloud resources.
Expand Down Expand Up @@ -168,24 +182,31 @@ Third, run the restore subcommand:
go run main.go restore --configuration examples/oss/configuration.yaml --state examples/oss/state.json
```

Finally, you can check the status of the configuration restored just now:
Then, you will see the output of the command like the flowing:

```text
2022/05/27 00:01:02 the Terraform backend was restored successfully
2022/05/27 00:01:02 try to restore the configuration......
2022/05/27 00:01:02 apply the configuration successfully, wait it to be available......
2022/05/27 00:01:02 the state of configuration is , wait it to be available......
2022/05/27 00:01:04 the state of configuration is ProvisioningAndChecking, wait it to be available......
2022/05/27 00:01:06 the state of configuration is ProvisioningAndChecking, wait it to be available......
2022/05/27 00:01:08 the state of configuration is ProvisioningAndChecking, wait it to be available......
2022/05/27 00:01:10 the state of configuration is ProvisioningAndChecking, wait it to be available......
2022/05/27 00:01:12 the state of configuration is ProvisioningAndChecking, wait it to be available......
2022/05/27 00:01:14 the state of configuration is ProvisioningAndChecking, wait it to be available......
2022/05/27 00:01:16 the configuration is available now
2022/05/27 00:01:16 try to print the log of the `terraform apply`......
```shell
$ kubectl get configuration.terraform.core.oam.dev
NAME STATE AGE
alibaba-oss-bucket-hcl-restore-example Available 13m
```

And you can check the logs of the `terraform-executor` in the pod of the "terraform apply" job:

```shell
$ kubectl logs alibaba-oss-bucket-hcl-restore-example-apply--1-b29d6 terraform-executor
alicloud_oss_bucket.bucket-acl: Refreshing state... [id=restore-example]
─────────────────────────────────────────────────────────────────────────────
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.
Your configuration already matches the changes detected above. If you'd like
to update the Terraform state to match, create and apply a refresh-only plan:
terraform apply -refresh-only
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Expand All @@ -194,4 +215,6 @@ Outputs:
BUCKET_NAME = "restore-example.oss-cn-beijing.aliyuncs.com"
```

You can see the "No changes.". This shows that we did not recreate cloud resources during the restore process.
The output is very clear, you can see the configuration is available.

And, you can see the `No changes.` in the log of the `terraform apply`. This shows that we did not recreate cloud resources during the restore process.
94 changes: 94 additions & 0 deletions hack/tool/backup_restore/cmd/backup.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
package cmd

import (
"context"
"log"
"os"

"github.com/oam-dev/terraform-controller/api/v1beta2"
"github.com/oam-dev/terraform-controller/controllers/configuration/backend"
"github.com/spf13/cobra"
v1 "k8s.io/api/core/v1"
"k8s.io/cli-runtime/pkg/genericclioptions"
"sigs.k8s.io/controller-runtime/pkg/client"
)

var configurationName string

// newBackupCmd represents the backup command
func newBackupCmd(kubeFlags *genericclioptions.ConfigFlags) *cobra.Command {
restoreCmd := &cobra.Command{
Use: "backup",
PreRunE: func(cmd *cobra.Command, args []string) error {
if configurationName == "" {
log.Fatal("please provide the name of the configuration which need to be backed up")
}
return nil
},
RunE: func(cmd *cobra.Command, args []string) error {
err := buildK8SClient(kubeFlags)
if err != nil {
return err
}
return backup(context.Background())
},
}
restoreCmd.Flags().StringVar(&configurationName, "name", "", "the name of the configuration which needs to be backed up")
return restoreCmd
}

func backup(ctx context.Context) error {
configuration := &v1beta2.Configuration{}
if err := k8sClient.Get(ctx, client.ObjectKey{Name: configurationName, Namespace: currentNS}, configuration); err != nil {
return err
}

// backup the state
if err := backupTFState(ctx, configuration); err != nil {
log.Fatalf("back up the Terraform state failed: %s \n", err.Error())
}

// backup the configuration
serializer := buildSerializer()
f, err := os.Create("configuration.yaml")
if err != nil {
return err
}
defer f.Close()
if _, err := f.WriteString("apiVersion: terraform.core.oam.dev/v1beta2\nkind: Configuration\n"); err != nil {
return err
}
cleanUpConfiguration(configuration)
if err := serializer.Encode(configuration, f); err != nil {
return err
}
log.Println("back up the Terraform state to configuration.yaml")
return nil
}

func backupTFState(ctx context.Context, configuration *v1beta2.Configuration) error {
backendInterface, err := backend.ParseConfigurationBackend(configuration, k8sClient)
if err != nil {
return err
}
k8sBackend, ok := backendInterface.(*backend.K8SBackend)
if !ok {
log.Println("the configuration doesn't use the kubernetes backend, the Terraform state won't be backed up")
return nil
}

secret := &v1.Secret{}
if err := k8sClient.Get(ctx, client.ObjectKey{Name: getSecretName(k8sBackend), Namespace: currentNS}, secret); err != nil {
return err
}
stateData := string(secret.Data["tfstate"])
state, err := decompressTRState(stateData)
if err != nil {
return nil
}
if err := os.WriteFile("state.json", state, os.ModePerm); err != nil {
return err
}
log.Println("back up the Terraform state to state.json")
return nil
}
127 changes: 87 additions & 40 deletions hack/tool/backup_restore/cmd/restore.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,40 +17,37 @@ limitations under the License.
package cmd

import (
"bytes"
"compress/gzip"
"context"
"fmt"
"io"
"log"
"os"
"os/exec"
"path/filepath"
"time"

"github.com/oam-dev/terraform-controller/api/types"
"github.com/oam-dev/terraform-controller/api/v1beta2"
"github.com/oam-dev/terraform-controller/controllers/configuration/backend"
"github.com/spf13/cobra"
batchv1 "k8s.io/api/batch/v1"
v1 "k8s.io/api/core/v1"
kerrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/serializer/json"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/cli-runtime/pkg/genericclioptions"
"sigs.k8s.io/controller-runtime/pkg/client"
)

var (
stateJSONPath string
configurationPath string
k8sClient client.Client
)

// newRestoreCmd represents the restore command
func newRestoreCmd(kubeFlags *genericclioptions.ConfigFlags) *cobra.Command {
restoreCmd := &cobra.Command{
Use: "restore",
RunE: func(cmd *cobra.Command, args []string) error {
var err error
k8sClient, err = buildClientSet(kubeFlags)
err := buildK8SClient(kubeFlags)
if err != nil {
return err
}
Expand All @@ -68,16 +65,8 @@ func newRestoreCmd(kubeFlags *genericclioptions.ConfigFlags) *cobra.Command {
return restoreCmd
}

func buildClientSet(kubeFlags *genericclioptions.ConfigFlags) (client.Client, error) {
config, err := kubeFlags.ToRESTConfig()
if err != nil {
return nil, err
}
return client.New(config, client.Options{})
}

func restore(ctx context.Context) error {
configuration, err := getConfiguration()
configuration, err := decodeConfigurationFromYAML()
if err != nil {
return err
}
Expand All @@ -92,28 +81,99 @@ func restore(ctx context.Context) error {
}

// apply the configuration yaml
// FIXME (loheagn) use the restClient to do this
applyCmd := exec.Command("bash", "-c", fmt.Sprintf("kubectl apply -f %s", configurationPath))
if err := applyCmd.Run(); err != nil {
if err := applyConfiguration(ctx, configuration); err != nil {
return err
}
return nil
}

func getConfiguration() (*v1beta2.Configuration, error) {
func decodeConfigurationFromYAML() (*v1beta2.Configuration, error) {
configurationYamlBytes, err := os.ReadFile(configurationPath)
if err != nil {
return nil, err
}
configuration := &v1beta2.Configuration{}
scheme := runtime.NewScheme()
serializer := json.NewSerializerWithOptions(json.DefaultMetaFactory, scheme, scheme, json.SerializerOptions{Yaml: true})
serializer := buildSerializer()
if _, _, err := serializer.Decode(configurationYamlBytes, nil, configuration); err != nil {
return nil, err
}
if configuration.Namespace == "" {
configuration.Namespace = currentNS
}
cleanUpConfiguration(configuration)
return configuration, nil
}

func applyConfiguration(ctx context.Context, configuration *v1beta2.Configuration) error {
log.Println("try to restore the configuration......")

if err := k8sClient.Create(ctx, configuration); err != nil {
return err
}

log.Println("apply the configuration successfully, wait it to be available......")

errCh := make(chan error)
timeoutCtx, cancel := context.WithTimeout(ctx, 20*time.Minute)
defer cancel()
go func() {
gotConf := &v1beta2.Configuration{}
for {
if err := k8sClient.Get(ctx, client.ObjectKey{Name: configuration.Name, Namespace: configuration.Namespace}, gotConf); err != nil {
errCh <- err
return
}
if gotConf.Status.Apply.State != types.Available {
log.Printf("the state of configuration is %s, wait it to be available......\n", gotConf.Status.Apply.State)
time.Sleep(2 * time.Second)
} else {
log.Println("the configuration is available now")
break
}
}
// refresh the configuration
configuration = gotConf
errCh <- nil
}()
select {
case err := <-errCh:
if err != nil {
return err
}

case <-timeoutCtx.Done():
log.Fatal("timeout waiting the configuration is available")
}

log.Printf("try to print the log of the `terraform apply`......\n\n")
if err := printExecutorLog(ctx, configuration); err != nil {
log.Fatalf("print the log of `terraform apply` error: %s\n", err.Error())
}

return nil
}

func printExecutorLog(ctx context.Context, configuration *v1beta2.Configuration) error {
job := &batchv1.Job{}
if err := k8sClient.Get(ctx, client.ObjectKey{Name: configuration.Name + "-apply", Namespace: configuration.Namespace}, job); err != nil {
return err
}
podList, err := clientSet.CoreV1().Pods(configuration.Namespace).List(ctx, metav1.ListOptions{LabelSelector: labels.FormatLabels(job.Spec.Selector.MatchLabels)})
if err != nil {
return err
}
pod := podList.Items[0]
logReader, err := clientSet.CoreV1().Pods(configuration.Namespace).GetLogs(pod.Name, &v1.PodLogOptions{Container: "terraform-executor"}).Stream(ctx)
if err != nil {
return err
}
defer logReader.Close()
if _, err := io.Copy(os.Stdout, logReader); err != nil {
return err
}
return nil
}

func resumeK8SBackend(ctx context.Context, backendInterface backend.Backend) error {
k8sBackend, ok := backendInterface.(*backend.K8SBackend)
if !ok {
Expand Down Expand Up @@ -145,14 +205,14 @@ func resumeK8SBackend(ctx context.Context, backendInterface backend.Backend) err
gotSecret.Type = v1.SecretTypeOpaque
}

if err := k8sClient.Get(ctx, client.ObjectKey{Name: "tfstate-default-" + k8sBackend.SecretSuffix, Namespace: k8sBackend.SecretNS}, &gotSecret); err != nil {
if err := k8sClient.Get(ctx, client.ObjectKey{Name: getSecretName(k8sBackend), Namespace: k8sBackend.SecretNS}, &gotSecret); err != nil {
if !kerrors.IsNotFound(err) {
return err
}
// is not found, create the secret
gotSecret = v1.Secret{
ObjectMeta: metav1.ObjectMeta{
Name: "tfstate-default-" + k8sBackend.SecretSuffix,
Name: getSecretName(k8sBackend),
Namespace: k8sBackend.SecretNS,
},
Type: v1.SecretTypeOpaque,
Expand All @@ -172,19 +232,6 @@ func resumeK8SBackend(ctx context.Context, backendInterface backend.Backend) err
return err
}
}
log.Println("the Terraform backend was restored successfully")
return nil
}

func compressedTFState() ([]byte, error) {
srcBytes, err := os.ReadFile(stateJSONPath)
var buf bytes.Buffer
writer := gzip.NewWriter(&buf)
_, err = writer.Write(srcBytes)
if err != nil {
return nil, err
}
if err := writer.Close(); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
1 change: 1 addition & 0 deletions hack/tool/backup_restore/cmd/root.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ func Execute() {
kubeConfigFlags := genericclioptions.NewConfigFlags(true)
kubeConfigFlags.AddFlags(rootCmd.PersistentFlags())
rootCmd.AddCommand(newRestoreCmd(kubeConfigFlags))
rootCmd.AddCommand(newBackupCmd(kubeConfigFlags))
err := rootCmd.Execute()
if err != nil {
os.Exit(1)
Expand Down
Loading

0 comments on commit ae8296e

Please sign in to comment.