From 47957ab357859bca9c941f4e68ac21e4be1aa786 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Tue, 16 Apr 2024 13:37:17 +0200 Subject: [PATCH 01/18] Add standard for volume backup functionality Signed-off-by: Markus Hentsch --- .../scs-XXXX-v1-volume-backup-service.md | 69 +++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 Standards/scs-XXXX-v1-volume-backup-service.md diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md new file mode 100644 index 000000000..297677481 --- /dev/null +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -0,0 +1,69 @@ +--- +title: Volume Backup Functionality +type: Standard # | Decision Record | Procedural +status: Draft +track: IaaS # | IaaS | Ops | KaaS | IAM +--- + +## Introduction + +OpenStack offers a variety of resources where users are able to transfer and store data in the infrastructure. +A prime example of these resources are volumes which are attached to virtual machines as virtual block storage devices. +As such they carry potentially large amounts of user data which is constantly changing at runtime. +It is important for users to have the ability to create backups of this data in a reliable and effifcient manner. + +## Motivation + +The [volume backup functionality of OpenStack](https://docs.openstack.org/cinder/latest/admin/volume-backups.html) is a feature that is not available in all OpenStack clouds per default. +The feature requires a backend to prepared and configured correctly before it can be used. +In Cinder this is a separate configuration to the general storage backend of the volume service and is not mandatory. +Thus, an arbitrary OpenStack cloud may or may not offer this feature. + +This standard aims to make this functionality the default in SCS clouds so that customers can expect the feature to be usable. + +## Design Considerations + +The standard should make sure that the feature is available and usable but should not limit the exact implementation (e.g. choice of backend driver) any further than necessary. + +### Options considered + +#### Only recommend volume backup feature, use images as alternative + +As an alternative to the volume backup feature of the Block Storage API, Glance images can also be created based on volumes and act as a backup under certain circumstances. +As an option, this standard could keep the actual integration of the volume backup feature optional and guide users how to use images as backup targets instead in case the feature is unavailable. + +However, it is not guaranteed that the Glance backend storage is separate from the volume storage. +For instance, both could be using the same Ceph cluster. +In such case, the images would not count as genuine backups. + +Although users are able to download images and transfer them to a different storage location, this approach might also prove unfeasible depending on the image size and the existence (or lack) of appropriate target storage on the user side. + +Furthermore, incremental backups are not possible when creating Glance images from volumes either. +This results in time-consuming backup operations of fully copying a volume everytime a backup is created. + +#### Mandate the availability of the volume backup feature + +This option is pretty straightforward. +It would make the volume backup feature mandatory for SCS clouds. +This way users can expect the feature to be available and usable. + +With this, users can leverage functionalities like incremental backups and benefit from optimized performance of the backup process due to the tight integration with the volume service. + +## Decision + +This standard decides to go with the second option and makes the volume backup feature mandatory in the following way: + +In an SCS cloud, the volume backup functionality MUST be configured properly and its API as defined per `/v3/{project_id}/backups` MUST be offered to customers. +If using Cinder, a suitable [backup driver](https://docs.openstack.org/cinder/latest/configuration/block-storage/backup-drivers.html) MUST be set up. + +The volume backup target storage SHOULD be a separate storage system from the one used for volumes themselves. + +## Related Documents + +- [OpenStack Block Storage v3 Backup API reference](https://docs.openstack.org/api-ref/block-storage/v3/index.html#backups-backups) + +## Conformance Tests + +Conformance tests include using the `/v3/{project_id}/backups` Block Storage API endpoint to create a volume and a backup of it as a non-admin user and subsequently restore the backup on a new volume while verifying the success of each operation. + + From 8e5eb04f2cbc27960f7f59d42c35286d910080ea Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Tue, 16 Apr 2024 16:21:43 +0200 Subject: [PATCH 02/18] Add test script to check volume backup API Signed-off-by: Markus Hentsch --- .../volume-backup/volume-backup-tester.py | 241 ++++++++++++++++++ 1 file changed, 241 insertions(+) create mode 100644 Tests/iaas/volume-backup/volume-backup-tester.py diff --git a/Tests/iaas/volume-backup/volume-backup-tester.py b/Tests/iaas/volume-backup/volume-backup-tester.py new file mode 100644 index 000000000..4a0713e82 --- /dev/null +++ b/Tests/iaas/volume-backup/volume-backup-tester.py @@ -0,0 +1,241 @@ +"""Volume Backup API tester for Block Storage API + +This test script executes basic operations on the Block Storage API centered +around volume backups. Its purpose is to verify that the Volume Backup API is +available and working as expected using simple operations such as creating and +restoring volume backups. + +It verifies that a properly configured backup driver is present to the extent +that aforementioned operations succeed on the API level. It does not by any +means verify that the backup and restore procedures actual handle the data +correctly (it only uses empty volumes and does not look at data for the sake +of simplicity). +""" + +import argparse +import getpass +import os +import time +import typing + +import openstack + +# prefix to be included in the names of any Keystone resources created +# used by the cleanup routine to identify resources that can be safely deleted +DEFAULT_PREFIX = "scs-test-" + +# timeout in seconds for resource availability checks +# (e.g. a volume becoming available) +WAIT_TIMEOUT = 60 + + +def connect(cloud_name: str, password: typing.Optional[str] = None + ) -> openstack.connection.Connection: + """Create a connection to an OpenStack cloud + + :param string cloud_name: + The name of the configuration to load from clouds.yaml. + + :param string password: + Optional password override for the connection. + + :returns: openstack.connnection.Connection + """ + + if password: + return openstack.connect( + cloud=cloud_name, + password=password + ) + else: + return openstack.connect( + cloud=cloud_name, + ) + + +def test_backup(conn: openstack.connection.Connection, + prefix=DEFAULT_PREFIX, timeout=WAIT_TIMEOUT) -> None: + """Execute volume backup tests on the connection + + This will create an empty volume, a backup of that empty volume and then + attempt to restore the backup onto a new volume. + Purpose of these tests is to verify that the volume backup API is working + correctly. + """ + + # CREATE VOLUME + print("Creating volume ...") + volume = conn.block_storage.create_volume( + name=f"{prefix}volume", + size=1 + ) + assert volume is not None, ( + "Initial volume creation failed" + ) + volume_id = volume.id + assert conn.block_storage.get_volume(volume_id) is not None, ( + "Retrieving initial volume by ID failed" + ) + + print( + f"↳ waiting for volume with ID '{volume_id}' to reach status " + f"'available' ..." + ) + seconds_waited = 0 + while conn.block_storage.get_volume(volume_id).status != "available": + time.sleep(1.0) + seconds_waited += 1 + assert seconds_waited < timeout, ( + f"Timeout reached while waiting for volume to reach status " + f"'available' (volume id: {volume_id}) after {seconds_waited} " + f"seconds" + ) + print("Create empty volume: PASS") + + # CREATE BACKUP + print("Creating backup from volume ...") + backup = conn.block_storage.create_backup( + name=f"{prefix}volume-backup", + volume_id=volume_id + ) + assert backup is not None, ( + "Backup creation failed" + ) + backup_id = backup.id + assert conn.block_storage.get_backup(backup_id) is not None, ( + "Retrieving backup by ID failed" + ) + + print(f"↳ waiting for backup '{backup_id}' to become available ...") + seconds_waited = 0 + while conn.block_storage.get_backup(backup_id).status != "available": + time.sleep(1.0) + seconds_waited += 1 + assert seconds_waited < timeout, ( + f"Timeout reached while waiting for backup to reach status " + f"'available' (backup id: {backup_id}) after {seconds_waited} " + f"seconds" + ) + print("Create backup from volume: PASS") + + # RESTORE BACKUP + print("Restoring backup to volume ...") + restored_volume_name = f"{prefix}restored-backup" + conn.block_storage.restore_backup( + backup_id, + name=restored_volume_name + ) + + print( + f"↳ waiting for restoration target volume '{restored_volume_name}' " + f"to be created ..." + ) + seconds_waited = 0 + while conn.block_storage.find_volume(restored_volume_name) is None: + time.sleep(1.0) + seconds_waited += 1 + assert seconds_waited < timeout, ( + f"Timeout reached while waiting for restored volume to be created " + f"(volume name: {restored_volume_name}) after {seconds_waited} " + f"seconds" + ) + # wait for the volume restoration to finish + print( + f"↳ waiting for restoration target volume '{restored_volume_name}' " + f"to reach 'available' status ..." + ) + volume_id = conn.block_storage.find_volume(restored_volume_name).id + while conn.block_storage.get_volume(volume_id).status != "available": + time.sleep(1.0) + seconds_waited += 1 + assert seconds_waited < timeout, ( + f"Timeout reached while waiting for restored volume reach status " + f"'available' (volume id: {volume_id}) after {seconds_waited} " + f"seconds" + ) + print("Restore volume from backup: PASS") + + +def cleanup(conn: openstack.connection.Connection, prefix=DEFAULT_PREFIX): + """ + Looks up volume and volume backup resources matching the given prefix and + deletes them. + """ + print(f"\nPerforming cleanup for resources with the " + f"'{prefix}' prefix ...") + + backups = conn.block_storage.backups() + for backup in backups: + if backup.name.startswith(prefix): + print(f"↳ deleting volume backup '{backup.id}' ...") + conn.block_storage.delete_backup(backup.id) + + volumes = conn.block_storage.volumes() + for volume in volumes: + if volume.name.startswith(prefix): + print(f"↳ deleting volume '{volume.id}' ...") + conn.block_storage.delete_volume(volume.id) + + +def main(): + parser = argparse.ArgumentParser( + description="SCS Domain Manager Conformance Checker") + parser.add_argument( + "--os-cloud", type=str, + help="Name of the cloud from clouds.yaml, alternative " + "to the OS_CLOUD environment variable" + ) + parser.add_argument( + "--ask", + help="Ask for password interactively instead of reading it from the " + "clouds.yaml", + action="store_true" + ) + parser.add_argument( + "--debug", action="store_true", + help="Enable OpenStack SDK debug logging" + ) + parser.add_argument( + "--prefix", type=str, + default=DEFAULT_PREFIX, + help=f"OpenStack resource name prefix for all resources to be created " + f"and/or cleaned up by this script within the configured domains " + f"(default: '{DEFAULT_PREFIX}')" + ) + parser.add_argument( + "--timeout", type=int, + default=WAIT_TIMEOUT, + help=f"Timeout in seconds for operations waiting for resources to " + f"become available such as creating volumes and volume backups " + f"(default: '{WAIT_TIMEOUT}')" + ) + parser.add_argument( + "--cleanup-only", action="store_true", + help="Instead of executing tests, cleanup all resources " + "with the prefix specified via '--prefix' (or its default)" + ) + args = parser.parse_args() + openstack.enable_logging(debug=args.debug) + + # parse cloud name for lookup in clouds.yaml + cloud = os.environ.get("OS_CLOUD", None) + if args.os_cloud: + cloud = args.os_cloud + assert cloud, ( + "You need to have the OS_CLOUD environment variable set to your " + "cloud name or pass it via --os-cloud" + ) + conn = connect( + cloud, + password=getpass.getpass("Enter password: ") if args.ask else None + ) + if args.cleanup_only: + cleanup(conn, prefix=args.prefix) + else: + cleanup(conn, prefix=args.prefix) + test_backup(conn, prefix=args.prefix, timeout=args.timeout) + cleanup(conn, prefix=args.prefix) + + +if __name__ == "__main__": + main() From 1563b7c89ad964f0bbfaed3b301a487e180d68e6 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Tue, 16 Apr 2024 17:07:04 +0200 Subject: [PATCH 03/18] Add README for test script Signed-off-by: Markus Hentsch --- Tests/iaas/volume-backup/README.md | 70 ++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 Tests/iaas/volume-backup/README.md diff --git a/Tests/iaas/volume-backup/README.md b/Tests/iaas/volume-backup/README.md new file mode 100644 index 000000000..2b6cd4716 --- /dev/null +++ b/Tests/iaas/volume-backup/README.md @@ -0,0 +1,70 @@ +# Volume Backup API Test Suite + +## Test Environment Setup + +### Test Execution Environment + +> **NOTE:** The test execution procedure does not require cloud admin rights. + +To execute the test suite a valid cloud configuration for the OpenStack SDK in the shape of "`clouds.yaml`" is mandatory[^1]. +**The file is expected to be located in the current working directory where the test script is executed unless configured otherwise.** + +[^1]: [OpenStack Documentation: Configuring OpenStack SDK Applications](https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html) + +The test execution environment can be located on any system outside of the cloud infrastructure that has OpenStack API access. +Make sure that the API access is configured properly in "`clouds.yaml`". + +It is recommended to use a Python virtual environment[^2]. +Next, install the OpenStack SDK required by the test suite: + +```bash +pip3 install openstacksdk +``` + +Within this environment execute the test suite. + +[^2]: [Python 3 Documentation: Virtual Environments and Packages](https://docs.python.org/3/tutorial/venv.html) + +## Test Execution + +The test suite is executed as follows: + +```bash +python3 volume-backup-tester.py --os-cloud mycloud +``` + +As an alternative to "`--os-cloud`", the "`OS_CLOUD`" environment variable may be specified instead. +The parameter is used to look up the correct cloud configuration in "`clouds.yaml`". +For the example command above, this file should contain a `clouds.mycloud` section like this: + +```yaml +--- +clouds: + mycloud: + auth: + auth_url: ... + ... + ... +``` + +If the test suite fails and leaves test resources behind, the "`--cleanup-only`" flag may be used to delete those resources from the domains: + +```bash +python3 volume-backup-tester.py --os-cloud mycloud --cleanup-only +``` + +For any further options consult the output of "`python3 volume-backup-tester.py --help`". + +### Script Behavior & Test Results + +> **NOTE:** Before any execution of test batches, the script will automatically perform a cleanup of volumes and volume backups matching a special prefix (see the "`--prefix`" flag). +> This cleanup behavior is identical to "`--cleanup-only`". + +The script will print all cleanup actions and passed tests to `stdout`. + +If all tests pass, the script will return with an exit code of `0`. + +If any test fails, the script will halt, print the exact error to `stderr` and return with a non-zero exit code. + +In case of a failed test, cleanup is not performed automatically, allowing for manual inspection of the cloud state for debugging purposes. +Although unnecessary due to automatic cleanup upon next execution, you can manually trigger a cleanup using the "`--cleanup-only`" flag of this script. From db121598562f152614cb152a587b47a28780ca37 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Tue, 16 Apr 2024 17:11:19 +0200 Subject: [PATCH 04/18] Mention test suite in standard Signed-off-by: Markus Hentsch --- Standards/scs-XXXX-v1-volume-backup-service.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md index 297677481..e39aaba11 100644 --- a/Standards/scs-XXXX-v1-volume-backup-service.md +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -66,4 +66,6 @@ The volume backup target storage SHOULD be a separate storage system from the on Conformance tests include using the `/v3/{project_id}/backups` Block Storage API endpoint to create a volume and a backup of it as a non-admin user and subsequently restore the backup on a new volume while verifying the success of each operation. - +There is a test suite in [`volume-backup-tester.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/volume-backup/volume-backup-tester.py). +The test suite connects to the OpenStack API and executes basic operations using the volume backup API to verify that the functionality requested by the standard is available. +Please consult the associated [README.md](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/volume-backup/README.md) for detailed setup and testing instructions. From b91620cdbd4f3d0ee63353b124ad9c529be02839 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Tue, 16 Apr 2024 17:11:32 +0200 Subject: [PATCH 05/18] Degrade standard to proposal for now Signed-off-by: Markus Hentsch --- Standards/scs-XXXX-v1-volume-backup-service.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md index e39aaba11..9fc570493 100644 --- a/Standards/scs-XXXX-v1-volume-backup-service.md +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -1,8 +1,8 @@ --- title: Volume Backup Functionality -type: Standard # | Decision Record | Procedural -status: Draft -track: IaaS # | IaaS | Ops | KaaS | IAM +type: Standard +status: Proposal +track: IaaS --- ## Introduction From 73441e4079c97acb14260016c5c3203338b97f8b Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Mon, 22 Apr 2024 14:27:02 +0200 Subject: [PATCH 06/18] Test script: wait for resource status in cleanup Signed-off-by: Markus Hentsch --- .../volume-backup/volume-backup-tester.py | 35 ++++++++++++++++--- 1 file changed, 31 insertions(+), 4 deletions(-) diff --git a/Tests/iaas/volume-backup/volume-backup-tester.py b/Tests/iaas/volume-backup/volume-backup-tester.py index 4a0713e82..2a4f37284 100644 --- a/Tests/iaas/volume-backup/volume-backup-tester.py +++ b/Tests/iaas/volume-backup/volume-backup-tester.py @@ -156,23 +156,50 @@ def test_backup(conn: openstack.connection.Connection, print("Restore volume from backup: PASS") -def cleanup(conn: openstack.connection.Connection, prefix=DEFAULT_PREFIX): +def cleanup(conn: openstack.connection.Connection, prefix=DEFAULT_PREFIX, + timeout=WAIT_TIMEOUT): """ Looks up volume and volume backup resources matching the given prefix and deletes them. """ + + def wait_for_resource(resource_type: str, resource_id: str, + expected_status="available") -> None: + seconds_waited = 0 + get_func = getattr(conn.block_storage, f"get_{resource_type}") + while get_func(resource_id).status != expected_status: + time.sleep(1.0) + seconds_waited += 1 + assert seconds_waited < timeout, ( + f"Timeout reached while waiting for {resource_type} during " + f"cleanup to be in status '{expected_status}' " + f"(id: {resource_id}) after {seconds_waited} seconds" + ) + print(f"\nPerforming cleanup for resources with the " f"'{prefix}' prefix ...") backups = conn.block_storage.backups() for backup in backups: if backup.name.startswith(prefix): + try: + wait_for_resource("backup", backup.id) + except openstack.exceptions.ResourceNotFound: + # if the resource has vanished on + # its own in the meantime ignore it + continue print(f"↳ deleting volume backup '{backup.id}' ...") conn.block_storage.delete_backup(backup.id) volumes = conn.block_storage.volumes() for volume in volumes: if volume.name.startswith(prefix): + try: + wait_for_resource("volume", volume.id) + except openstack.exceptions.ResourceNotFound: + # if the resource has vanished on + # its own in the meantime ignore it + continue print(f"↳ deleting volume '{volume.id}' ...") conn.block_storage.delete_volume(volume.id) @@ -230,11 +257,11 @@ def main(): password=getpass.getpass("Enter password: ") if args.ask else None ) if args.cleanup_only: - cleanup(conn, prefix=args.prefix) + cleanup(conn, prefix=args.prefix, timeout=args.timeout) else: - cleanup(conn, prefix=args.prefix) + cleanup(conn, prefix=args.prefix, timeout=args.timeout) test_backup(conn, prefix=args.prefix, timeout=args.timeout) - cleanup(conn, prefix=args.prefix) + cleanup(conn, prefix=args.prefix, timeout=args.timeout) if __name__ == "__main__": From 0beb9f1c55fa6c98c15b715197fb003b1b4d7ed8 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Mon, 22 Apr 2024 15:12:05 +0200 Subject: [PATCH 07/18] Test script: wait between interdependent cleanups Signed-off-by: Markus Hentsch --- Tests/iaas/volume-backup/volume-backup-tester.py | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/Tests/iaas/volume-backup/volume-backup-tester.py b/Tests/iaas/volume-backup/volume-backup-tester.py index 2a4f37284..b5de2a3e0 100644 --- a/Tests/iaas/volume-backup/volume-backup-tester.py +++ b/Tests/iaas/volume-backup/volume-backup-tester.py @@ -191,6 +191,19 @@ def wait_for_resource(resource_type: str, resource_id: str, print(f"↳ deleting volume backup '{backup.id}' ...") conn.block_storage.delete_backup(backup.id) + # wait for all backups to be cleaned up before attempting to remove volumes + seconds_waited = 0 + while len( + # list of all backups whose name starts with the prefix + [b for b in conn.block_storage.backups() if b.name.startswith(prefix)] + ) > 0: + time.sleep(1.0) + seconds_waited += 1 + assert seconds_waited < timeout, ( + f"Timeout reached while waiting for all backups with prefix " + f"'{prefix}' to finish deletion" + ) + volumes = conn.block_storage.volumes() for volume in volumes: if volume.name.startswith(prefix): From c51cf029f927f8165b2e7333cbfb7d6946eeb984 Mon Sep 17 00:00:00 2001 From: Markus Hentsch <129268441+markus-hentsch@users.noreply.github.com> Date: Tue, 23 Apr 2024 09:33:58 +0200 Subject: [PATCH 08/18] Update Standards/scs-XXXX-v1-volume-backup-service.md Co-authored-by: josephineSei <128813814+josephineSei@users.noreply.github.com> Signed-off-by: Markus Hentsch <129268441+markus-hentsch@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-backup-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md index 9fc570493..44b59d94d 100644 --- a/Standards/scs-XXXX-v1-volume-backup-service.md +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -15,7 +15,7 @@ It is important for users to have the ability to create backups of this data in ## Motivation The [volume backup functionality of OpenStack](https://docs.openstack.org/cinder/latest/admin/volume-backups.html) is a feature that is not available in all OpenStack clouds per default. -The feature requires a backend to prepared and configured correctly before it can be used. +The feature requires a backend to be prepared and configured correctly before it can be used. In Cinder this is a separate configuration to the general storage backend of the volume service and is not mandatory. Thus, an arbitrary OpenStack cloud may or may not offer this feature. From 1fd8aaeb1209fb8a0063efcf5e67ccc2a880c6c8 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Tue, 23 Apr 2024 15:56:33 +0200 Subject: [PATCH 09/18] Test script: minor error message improvement Signed-off-by: Markus Hentsch --- Tests/iaas/volume-backup/volume-backup-tester.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/Tests/iaas/volume-backup/volume-backup-tester.py b/Tests/iaas/volume-backup/volume-backup-tester.py index b5de2a3e0..23e2ea42d 100644 --- a/Tests/iaas/volume-backup/volume-backup-tester.py +++ b/Tests/iaas/volume-backup/volume-backup-tester.py @@ -173,7 +173,8 @@ def wait_for_resource(resource_type: str, resource_id: str, assert seconds_waited < timeout, ( f"Timeout reached while waiting for {resource_type} during " f"cleanup to be in status '{expected_status}' " - f"(id: {resource_id}) after {seconds_waited} seconds" + f"({resource_type} id: {resource_id}) after {seconds_waited} " + f"seconds" ) print(f"\nPerforming cleanup for resources with the " From 2910844fb2a34c5129887774133caad2857fcc4a Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Mon, 29 Apr 2024 15:48:41 +0200 Subject: [PATCH 10/18] Test script: adjust CLI description Signed-off-by: Markus Hentsch --- Tests/iaas/volume-backup/volume-backup-tester.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Tests/iaas/volume-backup/volume-backup-tester.py b/Tests/iaas/volume-backup/volume-backup-tester.py index 23e2ea42d..f4fa9522d 100644 --- a/Tests/iaas/volume-backup/volume-backup-tester.py +++ b/Tests/iaas/volume-backup/volume-backup-tester.py @@ -220,7 +220,7 @@ def wait_for_resource(resource_type: str, resource_id: str, def main(): parser = argparse.ArgumentParser( - description="SCS Domain Manager Conformance Checker") + description="SCS Volume Backup API Conformance Checker") parser.add_argument( "--os-cloud", type=str, help="Name of the cloud from clouds.yaml, alternative " From 61b5454bf4189a8d66814b04cdcb7be39c1afc2c Mon Sep 17 00:00:00 2001 From: Markus Hentsch <129268441+markus-hentsch@users.noreply.github.com> Date: Thu, 27 Jun 2024 13:33:28 +0200 Subject: [PATCH 11/18] Apply review suggestion Co-authored-by: anjastrunk <119566837+anjastrunk@users.noreply.github.com> Signed-off-by: Markus Hentsch <129268441+markus-hentsch@users.noreply.github.com> --- Standards/scs-XXXX-v1-volume-backup-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md index 44b59d94d..f324df81d 100644 --- a/Standards/scs-XXXX-v1-volume-backup-service.md +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -16,7 +16,7 @@ It is important for users to have the ability to create backups of this data in The [volume backup functionality of OpenStack](https://docs.openstack.org/cinder/latest/admin/volume-backups.html) is a feature that is not available in all OpenStack clouds per default. The feature requires a backend to be prepared and configured correctly before it can be used. -In Cinder this is a separate configuration to the general storage backend of the volume service and is not mandatory. +In Cinder, this is a separate configuration to the general storage backend of the volume service and is not mandatory. Thus, an arbitrary OpenStack cloud may or may not offer this feature. This standard aims to make this functionality the default in SCS clouds so that customers can expect the feature to be usable. From 3a853e7e5fcfc87e2757ab126031c5e5dd2fd285 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Thu, 27 Jun 2024 16:49:10 +0200 Subject: [PATCH 12/18] Add new design consideration about storage backend choice Signed-off-by: Markus Hentsch --- .../scs-XXXX-v1-volume-backup-service.md | 20 ++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md index f324df81d..e60deb899 100644 --- a/Standards/scs-XXXX-v1-volume-backup-service.md +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -12,6 +12,12 @@ A prime example of these resources are volumes which are attached to virtual mac As such they carry potentially large amounts of user data which is constantly changing at runtime. It is important for users to have the ability to create backups of this data in a reliable and effifcient manner. +## Terminology + +| Term | Meaning | +|---|---| +| CSP | Cloud Service Provider, provider managing the OpenStack infrastructure | + ## Motivation The [volume backup functionality of OpenStack](https://docs.openstack.org/cinder/latest/admin/volume-backups.html) is a feature that is not available in all OpenStack clouds per default. @@ -41,7 +47,7 @@ Although users are able to download images and transfer them to a different stor Furthermore, incremental backups are not possible when creating Glance images from volumes either. This results in time-consuming backup operations of fully copying a volume everytime a backup is created. -#### Mandate the availability of the volume backup feature +#### Focus on feature availability, make feature mandatory This option is pretty straightforward. It would make the volume backup feature mandatory for SCS clouds. @@ -49,6 +55,18 @@ This way users can expect the feature to be available and usable. With this, users can leverage functionalities like incremental backups and benefit from optimized performance of the backup process due to the tight integration with the volume service. +However, it does not seem feasible to also mandate having a separate storage backend for volume backups at the same time due to potential infrastructure limitations at CSP-side making it hard or even impossible to offer. +As such, the actual benefit of backups in terms of reliability and security aspects would be questionable if a separate storage backend is not mandated and therefore not guaranteed. + +This approach would focus on feature availability rather than backup reliability. + +#### Focus on backup reliability, make separate backend mandatory + +As an alternative, the volume backup feature availability could be made optional but in case a CSP chooses to offer it, the standard would mandate a separate storage backend to be used for volume backups. +This way, failures of the volume storage backend would not directly impact the availability and safety of volume backups, making them actually live up to their name. + +In contrast to the above, this approach would focus on backup reliability rather than feature availability. + ## Decision This standard decides to go with the second option and makes the volume backup feature mandatory in the following way: From a485855700c910e063c0edda1374e8b4d292874d Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Fri, 28 Jun 2024 16:56:51 +0200 Subject: [PATCH 13/18] Add documentation link regarding backup drivers Signed-off-by: Markus Hentsch --- Standards/scs-XXXX-v1-volume-backup-service.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md index e60deb899..b57893708 100644 --- a/Standards/scs-XXXX-v1-volume-backup-service.md +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -79,6 +79,7 @@ The volume backup target storage SHOULD be a separate storage system from the on ## Related Documents - [OpenStack Block Storage v3 Backup API reference](https://docs.openstack.org/api-ref/block-storage/v3/index.html#backups-backups) +- [OpenStack Volume Backup Drivers](https://docs.openstack.org/cinder/latest/configuration/block-storage/backup-drivers.html) ## Conformance Tests From 6967778d2e3c92a14475647afab465b0f4b2c333 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Fri, 26 Jul 2024 17:12:31 +0200 Subject: [PATCH 14/18] Mention that tests don't cover the optional part Signed-off-by: Markus Hentsch --- Standards/scs-XXXX-v1-volume-backup-service.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md index b57893708..2e6d69b66 100644 --- a/Standards/scs-XXXX-v1-volume-backup-service.md +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -84,7 +84,11 @@ The volume backup target storage SHOULD be a separate storage system from the on ## Conformance Tests Conformance tests include using the `/v3/{project_id}/backups` Block Storage API endpoint to create a volume and a backup of it as a non-admin user and subsequently restore the backup on a new volume while verifying the success of each operation. +These tests verify the mandatory part of the standard: providing the Volume Backup API. There is a test suite in [`volume-backup-tester.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/volume-backup/volume-backup-tester.py). The test suite connects to the OpenStack API and executes basic operations using the volume backup API to verify that the functionality requested by the standard is available. Please consult the associated [README.md](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/volume-backup/README.md) for detailed setup and testing instructions. + +Note that these tests don't verify the optional part of the standard: providing a separate storage backend for Cinder volume backups. +This cannot be checked from outside of the infrastructure as it is an architectural property of the infrastructure itself and transparent to customers. From 9ba095b5ba1f074c8ccbb0f80dfa6cffe0ece123 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Mon, 19 Aug 2024 17:13:09 +0200 Subject: [PATCH 15/18] Align header naming with latest standards template Signed-off-by: Markus Hentsch --- Standards/scs-XXXX-v1-volume-backup-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md index 2e6d69b66..f2962582d 100644 --- a/Standards/scs-XXXX-v1-volume-backup-service.md +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -67,7 +67,7 @@ This way, failures of the volume storage backend would not directly impact the a In contrast to the above, this approach would focus on backup reliability rather than feature availability. -## Decision +## Standard This standard decides to go with the second option and makes the volume backup feature mandatory in the following way: From 6cb1cc0f5c35ff86bf4b6984c35dd04beb33a00d Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Mon, 2 Sep 2024 16:12:11 +0200 Subject: [PATCH 16/18] Address review comments Signed-off-by: Markus Hentsch --- Standards/scs-XXXX-v1-volume-backup-service.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-XXXX-v1-volume-backup-service.md index f2962582d..9a4e1aca7 100644 --- a/Standards/scs-XXXX-v1-volume-backup-service.md +++ b/Standards/scs-XXXX-v1-volume-backup-service.md @@ -17,13 +17,16 @@ It is important for users to have the ability to create backups of this data in | Term | Meaning | |---|---| | CSP | Cloud Service Provider, provider managing the OpenStack infrastructure | +| IaaS | Abbreviation for Infrastructure as a Service | +| Image | IaaS resource representing a snapshot of a block storage disk, can be used to create Volumes | +| Volume | IaaS resource representing a virtual block storage device that can be attached as a disk to virtual machines | ## Motivation -The [volume backup functionality of OpenStack](https://docs.openstack.org/cinder/latest/admin/volume-backups.html) is a feature that is not available in all OpenStack clouds per default. +The [volume backup functionality of the Block Storage API](https://docs.openstack.org/cinder/latest/admin/volume-backups.html) is a feature that is not available in all clouds per default, e.g., in OpenStack. The feature requires a backend to be prepared and configured correctly before it can be used. -In Cinder, this is a separate configuration to the general storage backend of the volume service and is not mandatory. -Thus, an arbitrary OpenStack cloud may or may not offer this feature. +In the Block Storage service, the backup storage backend is usually configured separately from the storage backend of the general volume service and may not be mandatory. +Thus, an arbitrary cloud may or may not offer the backup feature in the Block Storage API. This standard aims to make this functionality the default in SCS clouds so that customers can expect the feature to be usable. @@ -35,16 +38,16 @@ The standard should make sure that the feature is available and usable but shoul #### Only recommend volume backup feature, use images as alternative -As an alternative to the volume backup feature of the Block Storage API, Glance images can also be created based on volumes and act as a backup under certain circumstances. +As an alternative to the volume backup feature of the Block Storage API, images can also be created based on volumes and act as a backup under certain circumstances. As an option, this standard could keep the actual integration of the volume backup feature optional and guide users how to use images as backup targets instead in case the feature is unavailable. -However, it is not guaranteed that the Glance backend storage is separate from the volume storage. +However, it is not guaranteed that the image backend storage is separate from the volume storage. For instance, both could be using the same Ceph cluster. In such case, the images would not count as genuine backups. Although users are able to download images and transfer them to a different storage location, this approach might also prove unfeasible depending on the image size and the existence (or lack) of appropriate target storage on the user side. -Furthermore, incremental backups are not possible when creating Glance images from volumes either. +Furthermore, incremental backups are not possible when creating images from volumes either. This results in time-consuming backup operations of fully copying a volume everytime a backup is created. #### Focus on feature availability, make feature mandatory From 66cdd6a27136d1eb4884195bd518d4a5d0461618 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Tue, 17 Sep 2024 10:18:31 +0200 Subject: [PATCH 17/18] Assign standard number Signed-off-by: Markus Hentsch --- ...ume-backup-service.md => scs-0117-v1-volume-backup-service.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename Standards/{scs-XXXX-v1-volume-backup-service.md => scs-0117-v1-volume-backup-service.md} (100%) diff --git a/Standards/scs-XXXX-v1-volume-backup-service.md b/Standards/scs-0117-v1-volume-backup-service.md similarity index 100% rename from Standards/scs-XXXX-v1-volume-backup-service.md rename to Standards/scs-0117-v1-volume-backup-service.md From 125b3bb2713095e295d69a740c4a0e75a5790726 Mon Sep 17 00:00:00 2001 From: Markus Hentsch Date: Tue, 17 Sep 2024 10:19:22 +0200 Subject: [PATCH 18/18] Change to draft state Signed-off-by: Markus Hentsch --- Standards/scs-0117-v1-volume-backup-service.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Standards/scs-0117-v1-volume-backup-service.md b/Standards/scs-0117-v1-volume-backup-service.md index 9a4e1aca7..d272dfa05 100644 --- a/Standards/scs-0117-v1-volume-backup-service.md +++ b/Standards/scs-0117-v1-volume-backup-service.md @@ -1,7 +1,7 @@ --- title: Volume Backup Functionality type: Standard -status: Proposal +status: Draft track: IaaS ---