Rename the latest snapshots that have not been rotated yet

README.md

@@ -28,7 +28,6 @@ Additional features:
   - [Command line options](#command-line-options)
   - [Passing the API token](#passing-the-api-token)
   - [Creating and rotating snapshots in a cron job](#creating-and-rotating-snapshots-in-a-cron-job)
-  - [Snapshots seem to be missing after rotation](#snapshots-seem-to-be-missing-after-rotation)
 - [Running the script in a container](#running-the-script-in-a-container)
   - [Passing the configuration file to the container](#passing-the-configuration-file-to-the-container)
   - [Environment variables](#environment-variables)
@@ -149,9 +148,11 @@ for which `rotate-snapshots` is `true`.
 
 "Rotating" means that existing snapshots will be renamed
 according to `snapshot-name`, or will be deleted if they
-are not contained in any of the configured `quarter-hourly`, `hourly`, ... `yearly`
+are no longer contained in any of the configured `quarter-hourly`, `hourly`, ... `yearly`
 periods. Those settings determine for how many such periods the snapshots will be retained.
 
+New snapshots that are _not yet_ contained in any rotation period will be renamed but not deleted.
+
 Snapshots that have been protected are neither renamed nor deleted during rotation.
 Nevertheless, they are taken into account in the rotation process.
 
@@ -162,23 +163,24 @@ Rotating snapshots is controlled by the following rules:
 - The first period is the shortest period immediately preceding the instant of rotation.
 - Other periods immediately precede the next shorter periods without gaps.
 - If a period contains multiple snapshots then only the oldest one will be retained for that period.
-- Rotated snapshots are renamed according to the template `snapshot-name`. This allows the server name and 
+- New and rotated snapshots are (re)named according to the template `snapshot-name`. This allows the server name and 
   labels, the period, the snapshot timestamp and environment variables to become part of the snapshot name.  
 See section [Snapshot name templates](#snapshot-name-templates) below for details.
 
 
 ### Snapshot name templates
 
 `snapshot-name` must be a string and may contain [Python format strings](https://docs.python.org/3/library/string.html#format-string-syntax).
+Snapshot names should be unique but this is not a requirement.
 The following field names are available for formatting:
 
 | Field name      |                                   Type                                   | Rendered as                                                                                                                                                                                                                                                                      |
 |-----------------|:------------------------------------------------------------------------:|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | `server`        |                                  `str`                                   | Server name, same as `server` in the [configuration file](#creating-the-configuration-file)                                                                                                                                                                                      |
 | `timestamp`     | [`datetime.datetime`](https://docs.python.org/3.8/library/datetime.html) | _Creation_ instant of the snapshot (_not changed by rotation_), expressed in the timezone of the system running this script. [`datetime`-specific formatting](https://docs.python.org/3.10/library/datetime.html#strftime-and-strptime-format-codes) may be used for this field. |
 | `label`         |                               `dict[str]`                                | Value of a server label at the _creation_ instant of the snapshot (_not changed by rotation_), may be referred to as e.g. `label[VERSION]`                                                                                                                                       |
-| `period_type`   |                                  `str`                                   | Type of period: `quarter-hourly`, `hourly`, ... `yearly`, or `latest` for a new snapshot that is not contained in any period                                                                                                                                                     |
-| `period_number` |                                  `int`                                   | Rotation number of the period: `1` = latest, `2` = next to latest and so on; always `0` for a new snapshot with period `latest`                                                                                                                                                  |
+| `period_type`   |                                  `str`                                   | Type of period: `quarter-hourly`, `hourly`, ... `yearly`, or `latest` for new snapshots that have not been rotated yet                                                                                                                                                           |
+| `period_number` |                                  `int`                                   | Rotation number of the period: `1` = latest, `2` = next to latest and so on; also applies to `latest` snapshots                                                                                                                                                                  |
 | `env`           |                               `dict[str]`                                | Value of an environment variable at the creation or rotation instant of the snapshot, may be referred to as e.g. `env[USER]`                                                                                                                                                     |
 
 
@@ -215,16 +217,10 @@ unauthorized access. The following methods are available for passing the API tok
 
 ### Creating and rotating snapshots in a cron job
 
-As a cron job, this script should run at least once per the shortest period for which snapshots are to be retained.  
-If, for example, the shortest retention period has been set to `daily` then the script should run at least daily.
-
-
-### Snapshots seem to be missing after rotation
+As a cron job, this script should run once per the shortest period for which snapshots are to be retained.  
+If, for example, the shortest retention period has been set to `daily` then the script should run daily.
 
-If several types of retention period (e.g. `daily`, `weekly` and `monthly`) have been defined then the latest snapshot
-will be contained in the latest period of each type at the same time.  
-Since the snapshot will be named after the longest period (`monthly`), there will not be any snapshots named 
-after the latest ones of the shorter retention periods (`daily` and `weekly`).
+If the script is run more frequently then several `latest` snapshots will be preserved. 
 
 
 ## Running the script in a container

src/hetzner_snap_and_rotate/__main__.py

@@ -2,7 +2,7 @@
 
 from datetime import datetime, timezone
 from syslog import LOG_DEBUG, LOG_ERR
-from typing import Dict, Tuple
+from typing import Dict, Tuple, Optional
 
 from hetzner_snap_and_rotate.config import Config
 from hetzner_snap_and_rotate.logger import log
@@ -11,20 +11,21 @@
 from hetzner_snap_and_rotate.snapshots import Snapshots, create_snapshot, Snapshot
 
 
-Rotated = Dict[Snapshot, Tuple[Period, int]]
+# Associates snapshots with their Period type ('latest' if the Period is None) and number
+Rotated = Dict[Snapshot, Tuple[Optional[Period], int]]
 
 
 def rotate(config: Config.Defaults, not_rotated: list[Snapshot], p_end: datetime) -> Rotated:
     rotated: Rotated = {}
-    at_start_of_period = False
+    latest_start = None
 
     for p in Period:
         p_count = getattr(config, p.config_name, 0) or 0
 
         if p_count > 0:
-            if not at_start_of_period:
-                p_end = p.start_of_period(p_end)
-                at_start_of_period = True
+            if latest_start is None:
+                latest_start = p.start_of_period(p_end)
+                p_end = latest_start
 
             for p_num, p_start in enumerate(p.previous_periods(p_end, p_count), start=1):
                 p_sn = Snapshots.oldest(p_start, p_end, not_rotated)
@@ -35,6 +36,12 @@ def rotate(config: Config.Defaults, not_rotated: list[Snapshot], p_end: datetime
 
                     p_end = p_start
 
+    # Assign numbers (but no period types) to the latest snapshots,
+    # or to all snapshots if no rotation period was configured
+    for l_num, l_sn in enumerate(Snapshots.latest(latest_start, not_rotated), start=1):
+        not_rotated.remove(l_sn)
+        rotated[l_sn] = (None, l_num)
+
     return rotated
 
 
@@ -60,7 +67,7 @@ def main() -> int:
 
                         new_snapshot = create_snapshot(srv, srv.config.snapshot_timeout)
 
-                    # If an exception occurred during powering down or snapshotting
+                    # If an exception occurred during powering down or taking the snapshot
                     # then throw it only after having restarted the server, if necessary
                     except Exception as ex:
                         caught = ex
@@ -82,9 +89,6 @@ def main() -> int:
                     # and note the new rotation period they are now associated with
                     not_rotated: list[Snapshot] = list(srv.snapshots)
 
-                    # Always keep the snapshot that has just been created
-                    not_rotated.remove(new_snapshot)
-
                     p_end = new_snapshot.created if new_snapshot is not None else datetime.now(tz=timezone.utc)
                     rotated = rotate(config=srv.config, not_rotated=not_rotated, p_end=p_end)
 

src/hetzner_snap_and_rotate/__version__.py

@@ -1 +1 @@
-__version__ = '1.1.1'
+__version__ = '1.1.2'

src/hetzner_snap_and_rotate/snapshots.py

@@ -129,7 +129,9 @@ def create_snapshot(server: Server, timeout: int = 300) -> Snapshot:
     if not config.dry_run:
         wrapper = server.perform_action(ServerAction.CREATE_IMAGE, return_type=SnapshotWrapper,
                                         data=data, timeout=timeout)
+        wrapper.image.created_from = server
         server.snapshots.append(wrapper.image)
+
         log(f'Server [{server.name}]: snapshot [{description}] has been created', LOG_INFO)
         return wrapper.image
 
@@ -139,7 +141,8 @@ def create_snapshot(server: Server, timeout: int = 300) -> Snapshot:
             description=description,
             protection=Protection(delete=False),
             created=datetime.now(tz=timezone.utc),
-            created_from=server
+            created_from=server,
+            labels=server.labels
         )
         server.snapshots.append(snapshot)
         return snapshot
@@ -170,3 +173,14 @@ def oldest(start: datetime, end: datetime, snapshots: list[Snapshot]) -> Optiona
         )
 
         return matching[0] if len(matching) else None
+
+    @staticmethod
+    def latest(start: Optional[datetime], snapshots: list[Snapshot]) -> list[Snapshot]:
+        predicate = (lambda s: s.created >= start) if start is not None else (lambda s: True)
+        matching = sorted(
+            filter(predicate, snapshots),
+            key=lambda s: s.created,
+            reverse=True
+        )
+
+        return matching