Stats: backup events indices and migration strategies

[ntarocco]

Stats backup

The view/download events are stored in monthly indices in the search cluster, and there is no backup out-of-the-box.

Possible solutions:

1. Setup a cronjob to run elasticdump and export all events to a local storage
2. Provide a documentation on how to connect your own OpenSearch cluster directly to an S3 storage

More complex alternatives:

• have Invenio (invenio-files-rest) to regularly export events and add it to a (yet another) bucket

Migration strategies

It would be useful to provide documentation, guidance or best practices on how to migrate legacy statistics.

1. Does it make sense to migrate each raw event?
2. Should you only migrate aggregated stats (e.g. per day, per record)?

[monotasker]

I need to migrate some legacy stats in the next month or so, so I'm quite interested in the answers here. I had been assuming that I would need to create raw events in the invenio indices. But if it's viable to simply migrate the aggregated stats, and if there aren't negative side effects, I might prefer that approach. Has anyone actually tried each strategy? Are there potential pitfalls involved in just migrating the aggregations?

[ntarocco]

You can of course only create the aggregated stats, and set the bookmark after the last timestamp that you have imported, so that the automatic aggregation will start only from new events. However, be aware that you won't be able to re-aggregate old events if you don't have the raw ones. A backup is strongly recommended.

[monotasker]

Yeah, that's what I suspected. I don't like the idea of not being able to re-generate the aggregations. But I don't have events coming from the data source--only cumulative aggregate numbers. So I've been considering creating artificial events in the Invenio index that would support the imported aggregations, basically just making up dates. The timing of the aggregated events will be off for the pre-import period, so I'm going to have to create fake dates for the aggregations anyway. Since they're time-bounded.

[monotasker]

On the question of backups: we're using AWS opensearch service, so we're having it generate daily backup snapshots of the indices. That solves our backup needs for the time being. But we also want to have more permanent off-site (i.e., not on AWS data centres) backups made periodically. It's not a big deal to generate those with something like a cron job. But there'd be added convenience if invenio-files-rest were to generate file-system/s3 dumps on a configurable schedule.

This is one of those cases where a custom solution per-deployment isn't hard. But everything takes developer hours, and we (like many implementations) are always working with too few people and too much work. So "convenience" isn't necessarily a trivial issue for implementers. (On the other hand, a built-in solution costs Invenio developer-hours that might be better spent elsewhere.)

[lnielsen]

invenio-files-rest were to generate file-system/s3 dumps on a configurable schedule.

Just for my clarification - you're still only referring to backup of stats right? or do you mean that you would like of dump of the records in InvenioRDM in this case?

If it's the latter(i.e. export all records and their files to a dump), then it's related to #154

[monotasker]

No, I'm just thinking about backing up the stats. Thanks, though. The ability to dump all the records and their files would definitely be useful! But at the moment I'm just focused on stats and generating a periodic file-system backup of those search indices.

[zzacharo]

Hey @monotasker ,

We (CERN team) we opted for the Opensearch snapshot functionality.

At CERN, we have a central Opensearch service that our cluster is running on. We started using the snapshot functionality, with a snapshot policy that is backing up the stats indices as per pattern <search-index-prefix>-events-stats* (search-index-prefix comes from the SEARCH_INDEX_PREFIX config variable.

These are backed up in s3-interface storage (not AWS) here at CERN. There you can setup a periodical job to snapshot the statistics events and in principle keeping a small number of past snapshots e.g <7 should be enough as in each snapshot you have all the events up to that day.

Additionally, the Opensearch service allows you to configure notifications for different events on the snapshot lifecycle e.g creation, deletion, failure which could be valuable assets to monitor the backup process.

[monotasker]

Thanks @zzacharo. We're on AWS services, and have been talking about using snapshots, so it's helpful to know how you approached it.