Persistence inconsstency in persistence.md, change request

[MarkusThur]

prestory, not that relevant, but how I came to it

I am using the latest stable release of openhabian 4.0.3 on a raspberry pi 3B+ 1GB. I'm using the default rrd4j persistance service. In the beginning all my items where bound to some kind of thing and I didn't have "virtual" items and everything went fine and as exspected. Then I introduced three items, storing time setpoints for some "user configurable" automations, like triggering a rule with Add Trigger >> Time Event >> at the time specified in an item's state. Also went fine for some time, but then I recognized, that whenever I have to restart the pi for whatever reason or it restarts by e.g. powerloss, those setpoints become NULL and do not recover thier previous state.
So I started to investigate into that issue and came to the idea, I may can recover them from rrd4j persistence and started reading forums and the documentation.
There I learned, that this may happen to "virtual items" and there are different sugestion for solutions to restoreOnStartup and some issues with it. Including, that it may doesn't make sense for all kinds of items to autorecover them on startup or that rules may trigger before the restore did happen and, and, and.. This also is reflected in the actual version of the documentation. But I found that the actual documention is not helpful in that regard.

issue in documentation, regarding especially the default rrd4j engine

The actual version jul 1 2023 states the default behaviour of the default installed rrd4j service differently:

# Persistence
[...]
openHAB ships with the [rrd4j persistence service](addons/persistence/rrd4j/) installed by default, which persists every Item on every state change and at least once a minute. Additionally, it restores the last stored value at system startup. To change this behaviour see [Persistence Configuration](#persistence-configuration).
[...]

so here: default is "everyChange, everyMinute, restoreOnStartup". As seen from my installation it most probably is not, at least for "unbound, virtual items" or it doesn't work as exspected.

later in the documentation is stated:

## Restoring Item States on Restart

When restarting your openHAB installation you may find there are times when your logs indicate some Items have the state `NULL`.
This is because, by default, Item states are not persisted when openHAB restarts - even if you have installed a persistence add-on.
In order for items to be persisted across openHAB restarts, you must define a `restoreOnStartup` strategy for your items.
Then whatever state they were in before the restart will be restored automatically.

this somehow describes what I observed.. My virtual Items with the time setpoints have the state NULL. And in contradiction to the general section, the default behaviour seems to be NOT to restore Items!!!, which means "real" items receive thier value with the first update from their binding and "virtual" items have to be somehow recovered by the "user".

Furthermore we learn, that the default behaviour of rrd4j service is NOT visible to the user in a rrd4j.persist file, which would be a follow up idea to make things easier. It would be really nice, if the default behaviour would not be somewhere in the implementation, but in that file, so one could see what is really going on and easyly do the whished change. ( I am aware that this is a own feature request somewhere else in this project)

then we have an example strategy file:

Strategies {
        everyHour : "0 0 * * * ?"
        everyDay  : "0 0 0 * * ?"
}

Items {
  item1, item2 : strategy = everyChange, restoreOnStartup
}

which is only somehow helpful... My suggestion would be to present somewhere representation of the default behaviour of rrd4j, (what it really does) which according to # persistance is most probably is something like following:

Strategies {
        everyMinute : "0 * * * * ?"
        everyDay  : "0 0 0 * * ?"
}

Filters {
  tenSeconds : T 10 s
}

Items {
  * : strategy = everyChange, everyMinute, restoreOnStartup filter = tenSeconds
}

or according to ## Restoring item states on restart is

Strategies {
        everyMinute : "0 * * * * ?"
}

Filters {
  tenSeconds : T 10 s
}

Items {
  * : strategy = everyChange, everyMinute filter = tenSeconds
}

but I fear it is even something else, what is the default :-). I asume, that it even doesn't do it every minute.

And at the Restore on startup section I then would like to present a .persist, that really makes sense, regarding the other information in it:

Strategies {
        everyMinute : "0 * * * * ?"
        everyHour : "0 0 * * * ?"
        everyDay  : "0 0 0 * * ?"
}
Filters {
  tenSeconds : T 10 s
}

Items {
  *  :  strategy = everyChange, everyMinute filter = tenSeconds
  unboundItem  :  strategy = restoreOnStartup
}

Furthermore I believe, the every minute strategy is also NOT the default behavior of a plain installation of rrdj4p, because this would make it somehow impossible to have an item that is logged only once / hour, unless u filter every change with T 60 min or something like that.

somehow the system needs also a way, to disable a strategy like stating -everyMinute, everyHour, which also is a request to the rules system, not the documentation :-).

change Request

So first of all, I request:

• to make the documentation consistent regarding the default behaviour of rrd4j persistence
• to let the documentation document the default behaviour of the software correctly
• to present the textual description also in a code sample, that refelcts the default behaviour, so that someone that whats a slightly modified behaviour can take this as a starting point
• after that starting requests to the persistence development to do default config .persistence, instead of hardcoded to the software

Thank you for your time reading the request :-)

[MarkusThur]

little update.. I investigated a bit more in it.. figuring out, that rrd4j is not able to persist datetime items.... So this request changes, to mention that the default persistance service is not able to do so :-).
As well as it still stays, that the description of what rrd4j does or does not is inconsistent and it's 100% unclear, what it does in default... Especially from what I further investigated, it seems that it does do restoreOnStartup as default, except for items, that it can't.. So even the standard setup may should be different and a change request to the software, as this is at least confusing behaviour.. But first of all, definitively this documentation has to be changed to represent the actual state

[MarkusThur]

I could reach a pretty decent behaviour regarding this by:

• installing MapDB add-on

configuring rrd4j.persist to:

Strategies {
        everyMinute : "0 * * * * ?"
}

Filters {
  tenSeconds : T 10 s
}

Items {
  * : strategy = everyChange, everyMinute filter = tenSeconds
//  meltingUnitsReadyTime, KlebergeraetOffTime, klebergeraetOnTime : strategy = restoreOnStartup
}

and MapDB.persist to:

Strategies {
}

Filters {

}

Items {
  * : strategy = everyChange, restoreOnStartup 
}

This is still not the best possible thing to do, but definitely better standard config than what is now and can definitely be used to document and explain the persist system..

[rkoshak]

tl;dr:

• this page is meant to be generic and apply to all persistence add-ons; rrd4j specific stuff does not belong here
• the original problem was an already documented limitation of rrd4j
• rrd4j is neither required nor is it (in OH 4 now) even installed by default; it's one option among many
• all documentation and other suggestions above belong on the rrd4j docs, not here, as all of them are rrd4j specific
• many of the suggested changes here already are documented in the rrd4j docs

There I learned, that this may happen to "virtual items" and there are different sugestion for solutions to restoreOnStartup and some issues with it.

Please point out where you saw this because Persistence does not even know whether an Item is "virtual" or not.

The fact that your "virtual" Items were not saved is something else and not a problem with the documentation. Virtual or linked Items should behave the same.

Furthermore we learn, that the default behaviour of rrd4j service is NOT visible to the user in a rrd4j.persist file, which would be a follow up idea to make things easier.

The default behavior is what happens when there is no .persist file.

then we have an example strategy file:
...
which is only somehow helpful... My suggestion would be to present somewhere representation of the default behaviour of rrd4j, (what it really does) which according to # persistance is most probably is something like following:

That first example looks like the example from the Persistence page. That's a universal example, not specific to rrd4j. And you'd only need to create a .persist file if you wanted to deviate from the default. I'm not sure what value is added by showing as an example a config file that does the same thing as when the .persist file is absent.

But, having said that, at least in rrd4j's case, the example .persist file on it's page does show the default behavior.'s config as a .persist file. https://www.openhab.org/addons/persistence/rrd4j/#examples

Strategies {
    // for rrd charts, we need a cron strategy
    everyMinute : "0 * * * * ?"
}

Items {
    // persist items on every change and every minute
    * : strategy = everyChange, everyMinute
}

Be careful here as the Persistence documentation page is generic and applies to all persistence add-ons. For specific behaviors and limitations of individual add-ons, see that add-ons' docs. We should not put rrd4j specific documentation in the generic Persistence page.

but I fear it is even something else, what is the default :-). I asume, that it even doesn't do it every minute.

Per the documentation, * stands for "all Items". That's your default behavior that will apply to all supported Items.

And at the Restore on startup section I then would like to present a .persist, that really makes sense, regarding the other information in it:

This is unnecessary. The * will include even unbound Items.

Furthermore I believe, the every minute strategy is also NOT the default behavior of a plain installation of rrdj4p, because this would make it somehow impossible to have an item that is logged only once / hour, unless u filter every change with T 60 min or something like that.

That's a technical limitation of how rrd4j works. If you know what you are doing you can change the period through the rrd4j.cfg file but by default you must save a value once every minute for each Item for rrd4j to work at all. If you make any other configuration you will not get any charts nor with any of the persistence operations work.

If you want to save only once every hour you either need to change the rrd4j.cfg to support that or you need to use some other database. If you only want to save on changes you must use some other database.

to make the documentation consistent regarding the default behaviour of rrd4j persistence

The rrd4j documentation is consistent with how rrd4j works. The generic Persistence documentation is consistent in-so-far as it describes the way all persistence add-ons work in OH.

to let the documentation document the default behaviour of the software correctly

The default behavior of rrd4j (and all the other persistence add-ons) are documented in each add-ons docs. This default behavior is usually the same but not always and it's implemented by each individual add-on. It does not belong in the generic Persistence docs.

to present the textual description also in a code sample, that refelcts the default behaviour, so that someone that whats a slightly modified behaviour can take this as a starting point

At least for rrd4j, that default strategy is shown in the docs.

after that starting requests to the persistence development to do default config .persistence, instead of hardcoded to the software

It was a deliberate design decision to allow persistence to work in a reasonable way without configuration. The primary (but not only) reason was to support the "Analyze" function in MainUI out-of-the-box.

little update.. I investigated a bit more in it.. figuring out, that rrd4j is not able to persist datetime items.... So this request changes, to mention that the default persistance service is not able to do so :-).

It is documented in the rrd4j docs:

NOTE: rrd4j is for storing numerical data only. It cannot store complex data types. The supported item types are therefore only Switch (internally mapped to 0/1), Dimmer, Number, Contact (internally mapped to 0/1), Rollershutter and Color (only brightness component stored).

#

This does not belong in the generic Persistence docs because it's specific to rrd4j.

Especially from what I further investigated, it seems that it does do restoreOnStartup as default, except for items, that it can't..

Obviously rrd4j can't restore Items it can't store.

But first of all, definitively this documentation has to be changed to represent the actual state

Any updates recommended thus far belong on the rrd4j add-on's docs, not here.

Other options include filing an issue to have rrd4j not restoreOnStartup by default. But that to is a different issue on a different repo.