Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Warn when newest snapshot is in the future #651

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

koleo
Copy link

@koleo koleo commented Jun 27, 2021

Description

Option --monitor-snapshots issues a Warning or a Critical when newest snapshot is too old based on the thresholds defined in the Sanoid configuration.

However, it reports an OK status when newest snapshot is in the future. This can happen when system time is wrong (in the future) for some time and then returns to normal. In such a situation, this may prevent the creation of new snapshots for a while and so we need to be alerted.

How to test

On a Linux Debian 10 server:

# Stop timesyncd
systemctl stop systemd-timesyncd.service

# Change date (+1 month)
date --set="$(date --date='next month')"

# take snapshots (they will be dated one month in the future)
./sanoid --take-snapshots

# Start timesyncd (the date will come back on time)
systemctl start systemd-timesyncd.service

# Monitor snapshots
## actual behaviour
./sanoid --monitor-snapshots ; echo $?
OK: all monitored datasets (tank/data2) have fresh snapshots
0
## new behaviour (this PR)
./sanoid --monitor-snapshots ; echo $?
WARN: tank/data2 newest daily snapshot is -29d -23h -56m -44s old (in the future), WARN: tank/data2 newest hourly snapshot is -29d -23h -56m -44s old (in the future), WARN: tank/data2 newest monthly snapshot is -4d -10h -29m -19s old (in the future)
1

Open questions

  • The decision to issue a Warning is arbitrary. It could also be a Critical.
  • We could emit a Warning when newest snapshot is in the near future, and a Critical when it's in the distant future, based on the same thresholds defined to detect recentness of the newest snapshots.
  • We could even add configuration options to define whether we want to issue a Warning, a Critical or just ignore such a situation. But that may be excessive for this infrequent event?

Option --monitor-snapshots issues a Warning or a Critical when newest
snapshot is too old based on the thresholds defined in the Sanoid
configuration.

However, it reports an _OK_ status when newest snapshot is in the
future. This can happen when system time is wrong (in the future) for
some time and then returns to normal. In such a situation, this may
prevent the creation of new snapshots for a while and so we need to be
alerted.

Signed-off-by: Nicolas Payart <[email protected]>
if (defined $snapsbytype{$path}{$type}{'newest'}) {
$elapsed = $snapsbytype{$path}{$type}{'newest'};
}
my $dispelapsed = displaytime($elapsed);
my $dispwarn = displaytime($warn);
my $dispcrit = displaytime($crit);
if ( $elapsed > $crit || $elapsed == -1) {
if ($elapsed > $crit || !defined($elapsed)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a potential undefined value is compared here and will probably trigger a warning

@@ -181,6 +181,10 @@ sub monitor_snapshots {
if (! $config{$section}{'monitor_dont_warn'} && ($errorlevel < 2) ) { $errorlevel = 1; }
push @msgs, "WARN: $path newest $type snapshot is $dispelapsed old (should be < $dispwarn)";
}
} elsif ($elapsed < 0) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a potential undefined value is compared here and will probably trigger a warning

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants