Skip to content

Restore Imager Service from a Backup

Jump to bottom
benoit74 edited this page Mar 21, 2024 · 3 revisions

If Imager Service is running

  • Enable maintenance mode on UI
    • set MAINTENANCE_MODE: "y" in cardshop/ui/ui-configs.cm.yaml (k8s)
    • apply
    • restart ui-deployment
  • Ensure no task is being run at https://imager.kiwix.org/scheduler/
  • Connect to worker ssh [email protected]
  • Shutdown worker docker stop imager-worker
  • backup API DB, UI DB and UI Medias by running the three cronjobs in cardshop
    • api-db-backup
    • ui-db-backup
    • ui-media-backup
  • check that backups went well on borgbase.com
  • Suspend all three cronjobs
  • Turn off API by scaling it to 0
  • Turn off API DB by scaling it to 0
  • Turn off UI DB by scaling it to 0

Restore API DB from borgbase

Backups are in borgbase. To download them you need the read-only credentials:

# those are all static values you need to enter
# those are all for the _slave_ (aka readonly) bitwarden account
export BW_CLIENTID=user.xxxxxxxxx
export BW_CLIENTSECRET=xxxxxxxxxxxx
export BW_PASSWORD=xxxxxxxxxxxx

Select a backup

docker run --rm -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name cardshop-scheduler-mongodb --list

cardshop-scheduler-mongodb is the name of the Borgbase repository in which we archive the Cardshop Scheduler DB backups.

Output would look like

List avaible archives ...
Remote: Warning: Permanently added the ECDSA host key for IP address '95.216.114.20' to the list of known hosts.
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] yes (from BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK)
cardshop-scheduler-mongodb__backup__2022-12-31T10:03:46 Sat, 2022-12-31 10:03:47 [c649da3f4a9597d368281ec0e767319b46f8a8d16b7e3e1787826b686729e874]
cardshop-scheduler-mongodb__backup__2023-03-31T10:03:44 Fri, 2023-03-31 10:03:46 [9128ae1f28117950549d5b92850c32dc887f781dc0116cb7bf3adb04e62a8337]
cardshop-scheduler-mongodb__backup__2023-04-30T10:03:45 Sun, 2023-04-30 10:03:46 [9c2a9658b1f5fac1e5d499466030622de212b67bd99e053dfca2d3021e231703]
cardshop-scheduler-mongodb__backup__2023-05-31T10:03:46 Wed, 2023-05-31 10:03:47 [0014e2c772994f0ea9e47276f56966f466f22278b7d02bcdba419b14aeaa778d]
cardshop-scheduler-mongodb__backup__2023-06-30T10:03:45 Fri, 2023-06-30 10:03:47 [5c7d0563777d8789e589dd97ca576f1c65ab06506f1b904899b7106fc1071582]
cardshop-scheduler-mongodb__backup__2023-07-31T10:03:45 Mon, 2023-07-31 10:03:46 [491cdc0167ac9d49b14024c6e6833404f116f3e0fb61a308d3cae3a43aecdfe1]
cardshop-scheduler-mongodb__backup__2023-08-31T10:03:48 Thu, 2023-08-31 10:03:49 [a4c11d18df91c89fcafb8182e69a33e3f56be842ae4b982769fdd67c54e89559]
cardshop-scheduler-mongodb__backup__2023-09-30T10:03:44 Sat, 2023-09-30 10:03:45 [552b3ef8944923f9b99549216c502040872629d771d6ee41baad25ed5f0c30ee]
cardshop-scheduler-mongodb__backup__2023-10-31T10:03:46 Tue, 2023-10-31 10:03:47 [156456edd5fa00e14a5b2b2f629ea95a6dc8d340a7749d3a50f8770dd2f38659]
cardshop-scheduler-mongodb__backup__2023-11-30T10:03:46 Thu, 2023-11-30 10:03:48 [1f77204a064813e253ea38eb8338edfeb8abc5c7a3160882faf1c793f75a1276]
cardshop-scheduler-mongodb__backup__2023-12-31T10:03:47 Sun, 2023-12-31 10:03:48 [1b3fc738f8d319d985fed70400f80a092a769828fa746dfc394a2b96687800a4]
cardshop-scheduler-mongodb__backup__2024-01-31T10:03:46 Wed, 2024-01-31 10:03:47 [98889feb9bb1aef916e1cf61d5c95f9be9cec48f01039adf8601039bf0691fd3]
cardshop-scheduler-mongodb__backup__2024-02-11T10:03:45 Sun, 2024-02-11 10:03:46 [e9c62d731d4a0660df9123ea11898e2d2b08c043e3b6a97dc35d86af60b84326]
cardshop-scheduler-mongodb__backup__2024-02-18T10:03:46 Sun, 2024-02-18 10:03:47 [c7451520592d1cac33e32822e219e0ae8f99fa6911244011bddf77bb312a7988]
cardshop-scheduler-mongodb__backup__2024-02-25T10:03:45 Sun, 2024-02-25 10:03:46 [a04efe4b546b39e3c7c5df154c144cc5096858f48a47a9e78950ec22d27ca314]
cardshop-scheduler-mongodb__backup__2024-02-29T10:03:45 Thu, 2024-02-29 10:03:47 [9d5206eac7fc8fbea5303c86a0b24711dc8202b2c06b2c9bc0ff742e97a7f0bc]
cardshop-scheduler-mongodb__backup__2024-03-03T10:03:45 Sun, 2024-03-03 10:03:46 [638391be0bebad64b75ddba3ec6be04a98486df13abaf886f552d9735e8fc605]
cardshop-scheduler-mongodb__backup__2024-03-10T10:03:46 Sun, 2024-03-10 10:03:47 [b3f4eb3712cef05ae4c70790dc89965468884b71c1e72c08038869db6a1b1361]
cardshop-scheduler-mongodb__backup__2024-03-15T10:03:46 Fri, 2024-03-15 10:03:47 [7d737d91083c90ce295e00aa396f2edd36d4d52fa5b636112b00c8705718b384]
cardshop-scheduler-mongodb__backup__2024-03-16T10:03:48 Sat, 2024-03-16 10:03:49 [976cf07af5bdd1b7c310152c3881d856d67bf2506e63160bab0e5daca652b568]
cardshop-scheduler-mongodb__backup__2024-03-17T10:03:49 Sun, 2024-03-17 10:03:50 [faf13466bcde3ba674a507e39822a22077178922704ee3a43d2f59abb414fcf8]
cardshop-scheduler-mongodb__backup__2024-03-18T10:03:50 Mon, 2024-03-18 10:03:51 [136222db84b5a90140d973e6cffcb39b4c6831432e41161744b57480767126ed]
cardshop-scheduler-mongodb__backup__2024-03-19T10:03:47 Tue, 2024-03-19 10:03:48 [9774a6af2b14d376e4c5ac147cd0f0f5ab534d95aea9f884ecff91634a069e8b]
cardshop-scheduler-mongodb__backup__2024-03-20T10:03:48 Wed, 2024-03-20 10:03:49 [76bd4e863bfeaaf254da58978d91240dfeb62475057a071850dc955b90e00091]

Choose one based on its date. Check the default backup periodicity in borg-backup tool and potential customization in k8s backup cronjob.

Note: the archive name is the first column (stops at first space). ex: cardshop-scheduler-mongodb__backup__2024-03-21T09:38:34.

Extract a Backup file

With your selected archive name, download+extract it to your filesystem:

docker run -v /data/restore:/restore:rw -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name cardshop-scheduler-mongodb --extract "cardshop-scheduler-mongodb__backup__2024-03-21T09:38:34"

Cardshop Scheduler backup is a single mongodump (bson) file that would be extracted to /data/restore in this example. The file has no extension

Test the dump file

# start a new mongo server
# note that this will create the credentials but not the Database
docker run --rm --name restore-mongo -it -p 27017:27017 -v $HOME/restore/:/data/db -e MONGO_INITDB_ROOT_PASSWORD=$MONGO_INITDB_ROOT_PASSWORD -e MONGO_INITDB_ROOT_USERNAME=$MONGO_INITDB_ROOT_USERNAME docker.io/mongo:4.2.9

# import the dump in the same container, still using the same credentials
docker exec -it restore-mongo mongorestore -j 1 --drop --preserveUUID -u $MONGO_INITDB_ROOT_USERNAME --password=$MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase=admin --authenticationMechanism=SCRAM-SHA-1 -d Cardshop --archive=/data/db/root/.borgmatic/mongodb_databases/api-db-service/Cardshop

Check the DB structure and data using any postgres client (Robo 3T)

Restore on k8s

  • Restore on API DB Volume using the borg accessor
  • Remove the accessor pod and job
  • Scale back API-DB to 1
  • restore using mongorestore by connecting to running API-db pod. Beware that this consumes a lot of RAM (x4 the dump size)
mongorestore -j 1 --drop --preserveUUID -u $MONGO_INITDB_ROOT_USERNAME --password=$MONGO_INITDB_ROOT_PASSWORD --authenticationDatabase=admin --authenticationMechanism=SCRAM-SHA-1 -d Cardshop --archive=/data/db/root/.borgmatic/mongodb_databases/api-db-service/Cardshop

Borg accessor

In order to get the dump file into the volume, one needs to launch borg-backup into the cluster. This would be done with a temporary Job

---
apiVersion: batch/v1
kind: Job
metadata:
  name: borg-accessor
  namespace: cardshop
spec:
  backoffLimit: 1
  template:
    metadata:
      labels:
        app: borg-app
    spec:
      containers:
      - name: borg-backup
        image: ghcr.io/kiwix/borg-backup
        command: ["restore", "--name", "cardshop-scheduler-mongodb", "--extract", "cardshop-scheduler-mongodb__backup__2024-03-21T09:38:34"]
        imagePullPolicy: Always
        env:
        - name: BW_CLIENTID
          value: "xx"
        - name: BW_CLIENTSECRET
          value: "xx"
        - name: BW_PASSWORD
          value: "xx"
        volumeMounts:
        - name: data-volume
          mountPath: "/restore"
          readOnly: false
      volumes:
      - name: data-volume
        persistentVolumeClaim:
          claimName: cardshop-api-db-pvc
      restartPolicy: Never
      nodeSelector:
        k8s.kiwix.org/role: "services"

Restore Manager media files from borgbase

Backups are in borgbase. To download them you need the read-only credentials:

# those are all static values you need to enter
# those are all for the _slave_ (aka readonly) bitwarden account
export BW_CLIENTID=user.xxxxxxxxx
export BW_CLIENTSECRET=xxxxxxxxxxxx
export BW_PASSWORD=xxxxxxxxxxxx

Select a backup

docker run --rm -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name cardshop-manager-media --list

cardshop-manager-media is the name of the Borgbase repository in which we archive the Cardshop Manager Media backups.

Output would look like

List avaible archives ...
Remote: Warning: Permanently added the ECDSA host key for IP address '95.216.113.224' to the list of known hosts.
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] yes (from BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK)
cardshop-manager-media__backup__2023-05-31T00:00:45 Wed, 2023-05-31 00:00:46 [c6f5684bd71f226e1ed92732bd339cd08d1242c3822f135ba651c23e7c3529ef]
cardshop-manager-media__backup__2023-06-30T00:00:52 Fri, 2023-06-30 00:00:53 [21cbf016f645be76626c165e6fda9e90bb92f2af492c7fbd4c9f6124431cea28]
cardshop-manager-media__backup__2023-07-31T00:01:04 Mon, 2023-07-31 00:01:06 [a4a583d8582e7141d7a162e33c119f1b55812d4d80aad2d0884f23fd73cd878d]
cardshop-manager-media__backup__2023-08-31T00:00:48 Thu, 2023-08-31 00:00:50 [3157494f3c985077472b4420447ade1aa3abf6042bb67f409120278ab1bee76e]
cardshop-manager-media__backup__2023-09-30T00:01:00 Sat, 2023-09-30 00:01:01 [1caea421f36db8384f10b629514f8b271bd34c22817b1411bc2bf3bfd23a9ef3]
cardshop-manager-media__backup__2023-10-31T00:01:03 Tue, 2023-10-31 00:01:04 [93059f13985a354a91b7ad24586bb4a32cb0102a7dcf08813ffdd59ab699fcf6]
cardshop-manager-media__backup__2023-11-30T00:00:56 Thu, 2023-11-30 00:00:57 [047ff65ac29a6cb943cf8028919136c93eec2cba3c73d01b5cdc7d70857a56c2]
cardshop-manager-media__backup__2023-12-31T00:00:50 Sun, 2023-12-31 00:00:51 [f00c466268cb5d695808f74f7469613c757d8ae93fbacf95484fddd631d9fe7e]
cardshop-manager-media__backup__2024-01-31T00:01:03 Wed, 2024-01-31 00:01:04 [d8b8f495050abccd8d5e94ceea3cc03e48a47b6c21a158584ad3dfe45e392a4f]
cardshop-manager-media__backup__2024-02-11T00:00:58 Sun, 2024-02-11 00:01:00 [11de3279322873e1da4173634aff5763fcf51dded324818767f9770a1aeb1dc3]
cardshop-manager-media__backup__2024-02-18T00:01:07 Sun, 2024-02-18 00:01:08 [caff9f9947ca2405c9a7c4a8670f1bbd3f7958ae084313f009ec5ed0c20e1070]
cardshop-manager-media__backup__2024-02-25T00:00:54 Sun, 2024-02-25 00:00:55 [0acf879e7d2a257bfff3b39234461564b0818bdeccf85ada55681e21f4622005]
cardshop-manager-media__backup__2024-02-29T00:01:01 Thu, 2024-02-29 00:01:02 [7e910c36470978febb78d3719ef290d6d8f43a09c940972c863de15ce668ab47]
cardshop-manager-media__backup__2024-03-03T00:00:52 Sun, 2024-03-03 00:00:53 [4dbd3c76a776fd6c55e339584d43690cf035485fbe050d6f7c09267e365ea77d]
cardshop-manager-media__backup__2024-03-10T00:01:14 Sun, 2024-03-10 00:01:15 [5c8139e73573156aeba7e1f9081c751d6b6dbd26de9c06ed89e6e29ab7dc73ae]
cardshop-manager-media__backup__2024-03-15T00:00:57 Fri, 2024-03-15 00:00:58 [d1b1ff1916632fc1c8e214ebbbca8882750dd991748ac77b453412d72565f42f]
cardshop-manager-media__backup__2024-03-16T00:00:56 Sat, 2024-03-16 00:00:57 [cc0549949d8a5bf0927687aa3ad47b97fb21322af92348b3cf2ac21fbbe4deb4]
cardshop-manager-media__backup__2024-03-17T00:00:52 Sun, 2024-03-17 00:00:54 [905d6d0c080c79ed794eb83db346373a552f378d2e957462d23fb5a7d120da6e]
cardshop-manager-media__backup__2024-03-18T00:01:09 Mon, 2024-03-18 00:01:10 [1ada7b24da0f31eac03cb7fb9a55368518a35186a40e7c6decccfff27d5af7d6]
cardshop-manager-media__backup__2024-03-19T00:00:52 Tue, 2024-03-19 00:00:53 [7f0222993c3206e06c6f2272fa22922a3f8a501a0c5a38ab0b927c155b90adc2]
cardshop-manager-media__backup__2024-03-20T00:01:01 Wed, 2024-03-20 00:01:02 [577b92fe1a3f67f5f68c82ea2c1893b0b43a50e6d62df15a5d7aeb6870039e07]
cardshop-manager-media__backup__2024-03-21T00:00:51 Thu, 2024-03-21 00:00:52 [cd49a14e2abe4c3d71d39aee6d59d87e2a224dba5c95a81154143dae52841368]
cardshop-manager-media__backup__2024-03-21T09:42:23 Thu, 2024-03-21 09:42:24 [5ace8ac92a3262a910d8dbfe248a8501b038825852676002463da764b64ee567]

Choose one based on its date. Check the default backup periodicity in borg-backup tool and potential customization in k8s backup cronjob.

Note: the archive name is the first column (stops at first space). ex: cardshop-manager-media__backup__2024-03-21T09:42:23.

Extract a Backup file

With your selected archive name, download+extract it to your filesystem:

docker run -v /data/restore:/restore:rw -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name cardshop-manager-media --extract "cardshop-manager-media__backup__2024-03-21T09:42:23"

Files are in storage/ subfolder of /restore.

Check that the files are present and OK!

Restore on k8s

  • Restore on UI Media Volume using the borg accessor
  • Remove the accessor pod and job

Borg accessor

In order to get the dump files into the volume, one needs to launch borg-backup into the cluster. This would be done with a temporary Job

---
apiVersion: batch/v1
kind: Job
metadata:
  name: borg-accessor
  namespace: cardshop
spec:
  backoffLimit: 1
  template:
    metadata:
      labels:
        app: borg-app
    spec:
      containers:
      - name: borg-backup
        image: ghcr.io/kiwix/borg-backup
        command: ["/bin/bash", "-c", "restore --name cardshop-manager-media --extract \"cardshop-manager-media__backup__2024-03-21T09:42:23\" && mv /restore/storage/* /restore/ && rmdir /restore/storage"]
        imagePullPolicy: Always
        env:
        - name: BW_CLIENTID
          value: "xx"
        - name: BW_CLIENTSECRET
          value: "xx"
        - name: BW_PASSWORD
          value: "xx"
        volumeMounts:
        - name: data-volume
          mountPath: "/restore"
          readOnly: false
      volumes:
      - name: data-volume
        persistentVolumeClaim:
          claimName: cardshop-ui-media-pvc
      restartPolicy: Never
      nodeSelector:
        k8s.kiwix.org/role: "services"

Restore UI DB from borgbase

Backups are in borgbase. To download them you need the read-only credentials:

# those are all static values you need to enter
# those are all for the _slave_ (aka readonly) bitwarden account
export BW_CLIENTID=user.xxxxxxxxx
export BW_CLIENTSECRET=xxxxxxxxxxxx
export BW_PASSWORD=xxxxxxxxxxxx

Select a backup

docker run --rm -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name cardshop-manager-mariadb --list

cardshop-manager-mariadb is the name of the Borgbase repository in which we archive the Cardshop UI DB backups.

Output would look like

List avaible archives ...
Remote: Warning: Permanently added the ECDSA host key for IP address '95.216.114.20' to the list of known hosts.
Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] yes (from BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK)
cardshop-manager-mariadb__backup__2022-12-31T00:00:49 Sat, 2022-12-31 00:00:50 [e0c3b4974c856fcfe412595b32c5da3f0a15d197b53cf16dd33c9ec12d34003d]
cardshop-manager-mariadb__backup__2023-03-31T00:00:50 Fri, 2023-03-31 00:00:51 [bbc398bb5dc73e7f06c7063b7f8f3572c8878da15accc52ced2861e5b01c3cd5]
cardshop-manager-mariadb__backup__2023-04-30T00:00:51 Sun, 2023-04-30 00:00:52 [13028c314eb720ee5cfcb78246a2593070f0f935fbbdc8479d65d9528e2841da]
cardshop-manager-mariadb__backup__2023-05-31T00:00:50 Wed, 2023-05-31 00:00:51 [5f74af650a4698a6ddfb3e31202f1af4e6973865cf107e6f043b4512e5e932b2]
cardshop-manager-mariadb__backup__2023-06-30T00:00:52 Fri, 2023-06-30 00:00:53 [e4905db31c552b62bab6f4a626887dca10e8ffc90e1a41cf8a6bdc4ebcf7b0b4]
cardshop-manager-mariadb__backup__2023-07-31T00:00:51 Mon, 2023-07-31 00:00:53 [c8aa60efc60ac5341aaf35d0223f449fcf4159b2ded6075c1a0db808184d0997]
cardshop-manager-mariadb__backup__2023-08-31T00:00:58 Thu, 2023-08-31 00:00:59 [e19fa2d47263a6b09e954fb73be298ad0bd9044bfc7959ac7b791fdec60f37c9]
cardshop-manager-mariadb__backup__2023-09-30T00:00:54 Sat, 2023-09-30 00:00:55 [6a98e449d913e50f2084695929793a2ac04368ee388a4ea112ca801cc84e35dc]
cardshop-manager-mariadb__backup__2023-10-31T00:00:53 Tue, 2023-10-31 00:00:54 [dc79976ba7af28a3518f520411ddcf4b6260198029b7ea3c1c9d54d3a31aa2b8]
cardshop-manager-mariadb__backup__2023-11-30T00:00:58 Thu, 2023-11-30 00:00:59 [eb7648d1576fa07954ccc5728ab510d9806d7ed38159485ad0b1e76948104301]
cardshop-manager-mariadb__backup__2023-12-31T00:00:56 Sun, 2023-12-31 00:00:57 [a40c9839d42b7a670809785fdd4fb296cfacf291f68aa2c3ea49a710df5f7b95]
cardshop-manager-mariadb__backup__2024-01-31T00:01:00 Wed, 2024-01-31 00:01:01 [ec006121d4d86189f4c73a8f8916e2c84799e1ee903609e3c72e6a9bc22fe2a9]
cardshop-manager-mariadb__backup__2024-02-11T00:00:55 Sun, 2024-02-11 00:00:56 [19a2870d2e8ea527c5a0afbe52d954092b908d465e8903938867f2fa4438f999]
cardshop-manager-mariadb__backup__2024-02-18T00:01:03 Sun, 2024-02-18 00:01:04 [603c934e5ae6cde7eda5aa73b6799485a3f40dfd92b7474b91b3d49fd96798af]
cardshop-manager-mariadb__backup__2024-02-25T00:00:57 Sun, 2024-02-25 00:00:58 [29558efac0cf744f62acb966ab227a2e0ba24e61401c3c2c1c76525a2b78000e]
cardshop-manager-mariadb__backup__2024-02-29T00:01:02 Thu, 2024-02-29 00:01:03 [59d474c93fdc7bfba50dc360bc0f6464a5b82658d022f3b52949dbfd1b48732f]
cardshop-manager-mariadb__backup__2024-03-03T00:01:02 Sun, 2024-03-03 00:01:03 [6f83e3f3df2d08e48438058521a6d6ffc9d715c73cfdadc2b2a8b446b81a6212]
cardshop-manager-mariadb__backup__2024-03-10T00:01:02 Sun, 2024-03-10 00:01:03 [feb75e8b739fc036530ae3765bb84f0eb9dbb1cbb80617210597fabaea1b154a]
cardshop-manager-mariadb__backup__2024-03-15T00:01:26 Fri, 2024-03-15 00:01:34 [85b1647486c21f13e7bf76365514fd9a489e39946dfca5a7154eb7bdbec9e909]
cardshop-manager-mariadb__backup__2024-03-16T00:00:56 Sat, 2024-03-16 00:00:57 [b710496ed22145f31ba2e672db431d4c3dfa626ac7cdddc2c7be5e4f1e0d22db]
cardshop-manager-mariadb__backup__2024-03-17T00:00:59 Sun, 2024-03-17 00:01:00 [32364b08272afc8a6232de706a7d5cbf3be9020dc1403bd062069bf634e81b7a]
cardshop-manager-mariadb__backup__2024-03-18T00:00:54 Mon, 2024-03-18 00:00:55 [017f9c4adbe0c39aba8b24e97fa0d3b36769021582764dfa38ffc0a9869a41ad]
cardshop-manager-mariadb__backup__2024-03-19T00:00:54 Tue, 2024-03-19 00:00:55 [9dd123b591e0228715be768171957a02c3b7b09e9c327a1b94c2dba60029cb30]
cardshop-manager-mariadb__backup__2024-03-20T00:01:20 Wed, 2024-03-20 00:01:21 [34fd909230fc1b98f0ae847ff1d5f219235fa2f259646c3fc6940c34bd02fe05]
cardshop-manager-mariadb__backup__2024-03-21T00:01:04 Thu, 2024-03-21 00:01:06 [cd07c20246852ebfdd559b2006cd2c9a69063ea251153bec8451861bf03818e0]
cardshop-manager-mariadb__backup__2024-03-21T09:42:22 Thu, 2024-03-21 09:42:24 [ebe854bfc34311267952c9e7c1274b2516b7c427ca2e665ebdd4ee855b3348bd]

Choose one based on its date. Check the default backup periodicity in borg-backup tool and potential customization in k8s backup cronjob.

Note: the archive name is the first column (stops at first space). ex: cardshop-manager-mariadb__backup__2024-03-21T09:42:22.

Extract a Backup file

With your selected archive name, download+extract it to your filesystem:

docker run -v /data/restore:/restore:rw -e BW_CLIENTID=$BW_CLIENTID -e BW_CLIENTSECRET=$BW_CLIENTSECRET -e BW_PASSWORD=$BW_PASSWORD ghcr.io/kiwix/borg-backup restore --name cardshop-manager-mariadb --extract "cardshop-manager-mariadb__backup__2024-03-21T09:42:22"

Cardshop UI backup is a single SQL Dump file that would be extracted to /data/restore in this example. The file has no extension ; move it to a more practical location

mv /data/restore/root/.borgmatic/postgresql_databases/api-postgres-db-service/zimfarm /data/restore/zimfarm

Test the dump file

# start a new mongo server
# note that this will create the credentials but not the Database
docker run --rm --name restore-mariadb -it -p 3306:3306 -v /data/restore:/restore -e MARIADB_DATABASE=$MARIADB_DATABASE -e MARIADB_USER=$MARIADB_USER -e MARIADB_PASSWORD=$MARIADB_PASSWORD -e MARIADB_ROOT_PASSWORD=$MARIADB_ROOT_PASSWORD  docker.io/library/mariadb:10.4 mysqld --max-allowed-packet=128M --innodb-buffer-pool-size=512M

# import the dump in the same container, still using the same credentials
docker exec -it restore-mariadb mysql --user=$MARIADB_USER --password=$MARIADB_PASSWORD --database $MARIADB_DATABASE < /restore/root/.borgmatic/mysql_databases/ui-db-service/cardshop

Check the DB structure and data using any mariadb client (DBeaver)

Restore on k8s

  • Restore on UI DB Volume using the borg accessor
  • Remove the accessor pod and job
  • Scale back UI-DB to 1
  • restore using mysql by connecting to running UI-db pod.
mysql --user=$MARIADB_USER --password=$MARIADB_PASSWORD --database $MARIADB_DATABASE < /var/lib/mysql/root/.borgmatic/mysql_databases/ui-db-service/cardshop

Borg accessor

In order to get the dump file into the volume, one needs to launch borg-backup into the cluster. This would be done with a temporary Job

---
apiVersion: batch/v1
kind: Job
metadata:
  name: borg-accessor
  namespace: cardshop
spec:
  backoffLimit: 1
  template:
    metadata:
      labels:
        app: borg-app
    spec:
      containers:
      - name: borg-backup
        image: ghcr.io/kiwix/borg-backup
        command: ["restore", "--name", "cardshop-manager-mariadb", "--extract", "cardshop-manager-mariadb__backup__2024-03-21T09:42:22"]
        imagePullPolicy: Always
        env:
        - name: BW_CLIENTID
          value: "xx"
        - name: BW_CLIENTSECRET
          value: "xx"
        - name: BW_PASSWORD
          value: "xx"
        volumeMounts:
        - name: data-volume
          mountPath: "/restore"
          readOnly: false
      volumes:
      - name: data-volume
        persistentVolumeClaim:
          claimName: cardshop-ui-db-pvc
      restartPolicy: Never
      nodeSelector:
        k8s.kiwix.org/role: "services"

Start the UI

  • Start (or restart) the ui-deployment
  • check logs to make sure it works as expected
  • disable MAINTENANCE_MODE
  • restart ui-deployment

Once everything's OK, resume the three cronjobs in cardshop

  • api-db-backup
  • ui-db-backup
  • ui-media-backup

Start a worker

Clone this wiki locally