Skip to content

Commit

Permalink
Add upgrading hints/warnings
Browse files Browse the repository at this point in the history
  • Loading branch information
yhabteab committed Mar 28, 2024
1 parent e8aa1ec commit 4633f26
Showing 1 changed file with 58 additions and 0 deletions.
58 changes: 58 additions & 0 deletions doc/04-Upgrading.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,64 @@
Specific version upgrades are described below. Please note that version upgrades are incremental.
If you are upgrading across multiple versions, make sure to follow the steps for each of them.

## Upgrading to Icinga DB v1.1.2

Please apply the `1.1.2.sql` upgrade script to your database. For package installations, you can find this file at
`/usr/share/icingadb/schema/mysql/upgrades/` or `/usr/share/icingadb/schema/pgsql/upgrades/`, depending on your
database vendor.

!!! warning

In order to fix a loading performance issue of the history view in Icinga DB Web, this upgrade script adds an
appropriate index on the `history` table. Creating this new index normally takes place without blocking any other
queries. However, there is this specific situation where some locking can go wrong and may hang for a relatively
considerable time, blocking all Icinga DB queries on all `*_history` tables and the `history` table inclusively if
there is an ongoing, long-running query on the `history` table. One way of causing this to happen is if an Icinga
Web user accesses the `icingadb/history` view at the exact moment you are running this script. Depending on how
many entries you have in the history table, Icinga DB Web may take quite a long time to load, until your web
servers timeout kicks in. See [unblock history tables](#unblock-history-tables) for details.

### Unblock History Tables

When you observe that the upgrade script has been taking an unusually long to complete, you can perform the following
analysis on another console and unblock it if necessary. It is important to note though that the script may need some
time to perform the reindexing on the `history` table even if it is not blocked. Nonetheless, you can use the
`show processlist` command to determine whether an excessive number of queries have been stuck in a waiting state.

```
MariaDB [icingadb]> show processlist;
+------+-----+-----+----------+-----+------+---------------------------------+------------------------------------+-----+
| Id | ... | ... | db | ... | Time | State | Info | ... |
+------+-----+-----+----------+-----+------+---------------------------------+------------------------------------+-----+
| 1475 | ... | ... | icingadb | ... | 1222 | Waiting for table metadata lock | INSERT INTO "notification_history" | ... |
| 1485 | ... | ... | icingadb | ... | 1262 | Creating sort index | SELECT history.id, history.... | ... |
| 1494 | ... | ... | icingadb | ... | 1224 | Waiting for table metadata lock | ALTER TABLE history ADD INDEX ... | ... |
| 1499 | ... | ... | icingadb | ... | 1215 | Waiting for table metadata lock | INSERT INTO "notification_history" | ... |
| 1500 | ... | ... | icingadb | ... | 1215 | Waiting for table metadata lock | INSERT INTO "state_history" ... | ... |
+------+-----+-----+----------+-----+------+---------------------------------+------------------------------------+-----+
39 rows in set (0.001 sec)
```

In the above output you can see that there are approximately `39` queries (`39 rows in set (0.001 sec)`), including the
`ALTER TABLE history ADD INDEX` query from the upgrade script, waiting for a metadata lock, they're just minimised to
the bare essentials. Unfortunately, only one of these queries is holding the `table metadata lock` that everyone else
is now waiting for, which in this case is a `SELECT` statement initiated by Icinga DB Web in the `icingadb/history`
view, which takes an unimaginably long time. Note that there might be multiple `SELECT` statements started before the
upgrade script in your case when the history view of your Icinga DB Web is opened by different Icinga Web users at the
same time.

You can now either just wait for the `SELECT` statements to finish by themselves and let them block the upgrade script
and all Icinga DB queries on all `*_history` tables or forcibly terminate them and let the remaining queries do their
work. In this case, cancelling that one blocking `SELECT` query will let the upgrade script continue normally without
blocking any other queries.
```
MariaDB [icingadb]> kill 1485;
```

In case you are insecure about which Icinga DB Web queries are blocking, you may simply cancel all `SELECT` statements
listed with `show processlist`. Cancelling a `SELECT` query is usually not that disastrous, so feel free to abort every
single one of them.

## Upgrading to Icinga DB v1.1.1

Please apply the `1.1.1.sql` upgrade script to your database.
Expand Down

0 comments on commit 4633f26

Please sign in to comment.