Enable a hot / cold storage model

[despens]

Hot storage are files currently in use and require to be quickly accessible. Nextcloud primary storage on local disk or block storage would be the best solution for that.

Cold storage is for files that are not in active use anymore and are treated like an archive. This data could be located on much cheaper storage offered by a specialized provider, or on slower storage media locally. In Nextcloud, you would want to use external storage for this case.

However, when files are moved in between different storage backends connected to a single Nextcloud instance, they lose their last-modified dates and appear as recently modified.

It is also not possible to use two Nextcloud instances with different primary storage connected to each, since transfer in between two instances via cloud federation also resets the last modified dates.

To enable the hot / cold storage model, files moved or copied in between in between different storage backends or Nextcloud instances need to retain all of their metadata, especially last modified information.

(This is an attempt to formulate a clearly bound use case for observations in #15192)

[SimplyCorbett]

Why can't you use cache for the hot data? Lots of options out there. http cache, kernel cache, filesystem cache.. etc. Works great for me.

[despens]

@dragon988 These options would not work at the scale that would make it economically meaningful to even distinguish in between hot and cold storage.

[SimplyCorbett]

@dragon988 These options would not work at the scale that would make it economically meaningful to even distinguish in between hot and cold storage.

Yes, it does. If you are scaling to enterprise level there are still options.

I'm not sure why you want nextcloud handling hot/cold storage. It's IMO a recipe for disaster.

[despens]

@dragon988 I just want to be able to move files from one storage back-end to another without losing file metadata. If that would exist, I could implement a storage model that fits my use-case. You seem to have found a solution using different caches that works for yours. Currently I am running a local file system cache in front of block storage, however the size of files I am dealing with makes caching less-than-ideal unfortunately. 🤷‍♀️

[solracsf]

Isn't this a duplicate of #15192 ?

[despens]

@acsfer As mentioned in the original issue, it is one specific effect of the bug described in #15192

[solracsf]

Okay, still agree with @dragon988 hot/cold should be a storage specific problem, not Nextcloud (IMHO).
You can achieve this (ex.) with some Object Storage providers and Lifecycle rules. Like https://www.scaleway.com/en/docs/object-storage-glacier/

[despens]

Everyone, thanks for your advice. At my organization we have considered all of these options and figured out we need something else, that we could build with Nextcloud if it supported data transfer in between storage backends. Since the original issue seemed to be too obscure, I tried to narrow it down to a use-case that I thought would be less abstract that "last modified metadata". Apparently these terms aren't the right way to talk about this. So I will close this issue.

[wiswedel]

@despens: Is this what you're looking for? nextcloud/files_retention#45 (however not available yet)

[despens]

Thank you @wiswedel, that sounds like a great formulation of what would be useful for me!