Write checkable create & delete sla history events

[yhabteab]

Why do we need this!

Currently we are generating the SLA history events only when e.g. there were state change and downtime start and end events for the checkables. Under some circumstances (if a checkable is created once and never deleted) this should be sufficient. However, when you e.g. delete a host and create it again a couple of days later and want to generate sla reports for this host at the end of the week, the result can vary depending on which state the host had before it was deleted. In order to be able to generate sla reports as accurately as possible, we decided to track the checkable creation and deletion time on top of the existing info. And since Icinga 2 doesn't really know when an object has been deleted (at least not in a simple way), this PR should take care of it.

Though, Icinga DB doesn't know when an object has been deleted either, it just takes the time the delete event for that object arrived and puts it into the new table. Meaning when you delete checkables while Icinga DB is stopped, the events Icinga DB would write after it is started won't reflect the actual delete/create event. Though, there is no better way to handle this gracefully.

Config sync

The upgrade script for 1.3.0 generates a created_at sla lifecycle entry for all existing hosts and services once it is applied as proposed in #566 (comment). Thus, all special cases such as the implementation of a custom fingerprint type for services¹ and performing an extra query to retrieve host IDs from the database for runtime deleted services, have been removed.

Implementation

The new table sla_history_lifecycle has a primary key over (id, delete_time) where delete_time=0 means "not deleted yet" (the column has to be NOT NULL due to being included in the primary key). id is either the service or host ID for which that sla lifecycle is being generated. This ensures that there can only be row per object that states that the object is currently alive in Icinga 2.

Initial sync

Icinga DB performs a simple INSERT statement for Host and Service types after each initial config dump unconditionally, but matches on hosts and services that don't already have a create_time entry with delete_time = 0 in the sla_lifecycle table, and sets their create_time timestamp to now. Additionally, it also updates the delete_time of each existing sla_lifecycle entries whose host/service IDs cannot be found in the Host/Service tables. It's unlikely, but when a given Checkable doesn't already have a create_time entry in the database, the update query won't update anything. Likewise, the insert statements may also become a no-op if the Checkables already have a create_time entry with delete_time = 0.

Create

Nothing to be done here (all newly created objects will be covered by the bulk INSERT ... SELECT FROM host/service queries after the config dump).

Update

Nothing to be done here (object existed before and continues to exist).

Delete

Nothing to be done here (all deleted objects will be covered by general bulk sea_lifecycle queries after the config dump).

Runtime updates

Upsert

Performs an INSERT with ignore for duplicate keys for both create and update events (these look identical in the runtime update stream). If the object is already marked as alive in sla_history_lifecycle, this will do nothing, otherwise it will mark it as created now (including when an object that was created before this feature was enabled is updated).

Delete

It assumes that there exists a created_at sla_lifecycle entry for that checkable currently going to be deleted, and performs a simple UPDATE statement setting delete_time = now (i.e. updates the PK of the row) marking the alive row for the object as deleted. If, for whatever reason, there is no corresponding created_at entry for this checkable, that update statement becomes a no-op, as the upgrade script and/or the initial config dump should have generated the necessary entries for all existing objects that were created before this feature was available.

Footnotes

1. Before @julianbrost proposed a change in https://github.com/Icinga/icingadb/pull/566#issuecomment-2273088195, services had to implement an additional custom fingerprint type ServiceFingerprint which was used to also retrieve their host IDs when computing the config delta of the initial sync. By introducing this type, the necessity of having to always perform an extra SELECT query to additionally retrieve the host IDs was eliminated, as host ID is always required for the sla lifecycles to work. ↩

[julianbrost]

I just had an idea how we could call that type of SLA history after we didn't really come up with good name for this initially: lifecycle

[Al2Klimov]

Please don’t force-push for now.

[oxzi]

I have played with this PR in a testing environment and have not encountered any errors so far.

However, I have a more general question: How will the sla_lifecycle table be used later on? The get_sla_ok_percent function does not honor it and, unless I am missing something, cannot even do this for deleted checkable objects.

[oxzi]

I was thinking about creating some indices for the queries within the SyncCheckablesSlaLifecycle function.

In a huge environment, those queries may take some time. However, I cannot say how huge they must become to make those queries slow or if Icinga 2 will be the performance bottleneck before.

I have not tested this or anything, just wanted to put the idea out there.

[julianbrost]

or if Icinga 2 will be the performance bottleneck before

If there's an issue, you could probably trigger it without Icinga 2 becoming an issue. You can create and delete many hosts and services over time, keeping the active config in Icinga 2 at a constant size but growing this table.

[oxzi]

Valid point, I have not considered cloud native fast moving infrastructures with a multitude of short-lived VMs.

[yhabteab]

The get_sla_ok_percent function does not honor it and, unless I am missing something, cannot even do this for deleted checkable objects.

That's right and it will never make use of it as well! As we discussed this last time, we will only collect the data for the time being until we get the time to finalise Icinga/icingadb-web#710 which will make use of this newly introduced table and delegate the SLA generation from SQL procedure to PHP code and mark the get_sla_ok_percent procedure as a feature freeze.

[yhabteab]

I've pushed the new transparent Screenshots and also changed the main function a bit, so I won't be pushing anything from now on unless someone requests a change.