Disaster Recovery Tool for Hyperledger Fabric on Kubernetes

I. Introdution

Akc-Horcrux is designed to backup and restore a Hyperledger Fabric network deployed on Kubernetes from potential catastrophic events. Like the dark lord, as long as there is a remaining horcrux, the network cannot die. To create a Hyperledger Fabric network using Kubernetes, we provide another tool call AKC-Mamba.

When running a Fabric blockchain network, we often run things in high availability mode. We might run multiple peers node and services, then if a peer dies, the other continues to function properly. Later, because of the Hyperledger Fabric mechanism, the administrator can always remove the faulty peer and create a new one to re-sync data back from other peer nodes.

So why does a peer die ?

Unfortunately, a peer is a piece of software in a very complex network of many components. In Hyperledger Fabric, we have a lot of problems of the Kafka-based ordering service. Sometimes, the peer or its state database data is corrupted due to disk faulty or malicious intention. Funny enough, we once had to deal with the case where the master administrator accidentally delete the whole production Kubernetes cluster for cost-saving purpose. After these kinds of event, the peer becomes faulty and just refuses to continue to accept blocks.

Great, we will create new peer and sync back data from the network.

This process consumes a lot of time as our ledger can only grow over time. The peer cannot just simply download the ledger from other peer nodes. It must, in fact, verify every block in the ledger from the genesis block until the most recent one.

To put it simpy, the good old practice of backup is the best way to protect network data. It is also useful for routine administrative purposes such as migrating cluster resources to other cluster, or replicating clusters for development and testing.

II. What do we need to backup?

In a Hyperledger Fabric network on Kubernetes, or simply any Kubernetes based solution, we need to backup two things:

• Configuration, State and Metadata of the cluster: resources are defined with a Kubernetes CRD (Custom Resource Definition) stored in etcd.
• Persistent volumes: includes Crypto materials, Channel artifacts, Peer backup files, Orderer backup files to recreate the network.

Obviously, we can't do all of these tasks by ourselves. We rely on a VMWare tool called velero to backup our Kubernetes cluster.

III. Hoxcrux tool

Features:

install	Setup environment for horcrux tool
backup	Create a backup, then store to cloud storage as configration in env.sh file
start-schedule-backup	Create a schedule backup
stop-schedule-backup [schedule_name]	Remove a schedule backup
restore [backup_name]	Create a restore
get-backups	Get backups
get-restores	Get restores
del-backup [backup_name]	Delete a backup
del-restore [restore_name]	Delete a restore
uninstall	Uninstall horcrux

IV. How to use?

1. Install

Description

To setup environment for hoxcrux tool and make alias hoxcrux for it.

To install velero client and velero server on K8S cluster.

If you want to change config for storage provisioner, you must run uninstall feature before use this.

Pre-requisites

• zsh/bash (zsh is prefer for automated completion)
• Configure storage provisioner of AWS in file env.sh
• A credential file for AWS: ~/.aws/credentials

Command

$ ./scripts/horcrux.sh install

2. Uninstall

Description

To remove resources of velero server on your K8S cluster

Command

$ horcrux uninstall

3. Backup

Description

Create a backup request for one or more namespaces, then store it to AWS.

Configuration, State and Metadata of the cluster are stored on S3 service. Persistent volumes are stored on snapshots service.

Pre-requisites

Configure BACKUP_NAME and BACKUP_NAMESPACE in env.sh file.

BACKUP_NAME must be unique. If BACKUP_NAMESPACE=all, then all namespaces will be backup.

BACKUP_NAME=sonchain-backup5
BACKUP_NAMESPACE=sonakachain,sonkafka,sonorderer,sonchain
# BACKUP_NAMESPACE=all

Command

$ horcrux backup

Output:

Backup request "sonchain-backup5" submitted successfully.
Run `velero backup describe sonchain-backup5` or `velero backup logs sonchain-backup5` for more details.

Check

Then, using get-backups feature to see your backups

$ horcrux get-backups

Output:

NAME                             STATUS                       CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
sonchain-backup                  Completed                    2020-03-31 03:31:01 +0000 UTC   21d       default            <none>
sonchain-backup1                 Completed                    2020-04-03 02:57:46 +0000 UTC   24d       default            <none>
sonchain-backup2                 Completed                    2020-04-03 03:46:07 +0000 UTC   24d       default            <none>
sonchain-backup3                 Completed                    2020-04-03 05:28:06 +0000 UTC   25d       default            <none>
sonchain-backup4                 Completed                    2020-04-03 07:32:37 +0000 UTC   25d       default            <none>
sonchain-backup5                 Completed                    2020-04-03 07:40:32 +0000 UTC   25d       default            <none>

Using velero backup describe [BACKUP_NAME] to see backup process and more details.

$ velero backup describe sonchain-backup5

Delete

$ horcrux del-backup sonchain-backup5

4. Restore

Description

Create a restore request from a BACKUP_NAME.

Command

You can use horcrux get-backups to get all BACKUP_NAMEs

$ horcrux restore sonchain-backup5

Check

$ horcrux get-restores

Output:

NAME                              BACKUP             STATUS      WARNINGS   ERRORS   CREATED                         SELECTOR
sonchain-backup2-20200403051836   sonchain-backup2   Completed   0          0        2020-04-03 05:18:37 +0000 UTC   <none>
sonchain-backup3-20200403053751   sonchain-backup3   Completed   0          0        2020-04-03 05:37:52 +0000 UTC   <none>
sonchain-backup5-20200403074443   sonchain-backup5   Completed   0          0        2020-04-03 07:44:44 +0000 UTC   <none>

Delete

$ horcrux del-restore sonchain-backup5-20200403074443

4. Schedule backup

Description

Create schedule backup

Pre-requisites

• Configure BACKUP_NAME and BACKUP_NAMESPACE in env.sh file.
• Configure SCHEDULE_TIME. Interval is specified by a Cron expression.

Command

$ horcrux start-schedule-backup

Check

Scheduled backups are saved with the name <SCHEDULE NAME>-<TIMESTAMP>, where is formatted as YYYYMMDDhhmmss.

$ horcrux get-backups

Output:

NAME                             STATUS                       CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
sonchain-backup-20200403062531   Completed                    2020-04-03 06:25:31 +0000 UTC   24d       default            <none>

Stop schedule backup

Using: horcrux stop-schedule-backup BACKUP_NAME

$ horcrux stop-schedule-backup sonchain-backup

V. LICENSE

Akc-Horcrux project source code files are made available under MIT license, located in the LICENSE file. Basically, you can do whatever you want as long as you include the original copyright and license notice in any copy of the software/source.