How does f11d deal with data version control?

[augusto-herrmann]

Background

When dealing with small data, using git for CSV is just fine. However, if the dataset begins to grow, or is updated too frequently, the overhead quickly starts to add up.

There have been many open source solutions for data version control, such as:

• dat
• dolt
• dvc
• probably others

Question

Should the f11d tool set cater to integrate to any, or all of those? Is it possible to have a data workflow that does not only continuous data integration but also version control? Should we care?

[rufuspollock]

@augusto-herrmann a great question 👏 😄

I've spent quite a bit of time recently thinking about data "versioning" and version control and am actively working on it, see e.g. http://tech.datopian.com/versioning/

To start with could you share a bit more about the underlying user/job stories for this e.g. are you concerned about efficient storage, efficient syncing, diffing, patching etc etc. You may also want to look at the features list I've already got at http://tech.datopian.com/versioning/#features

[augusto-herrmann]

I've spent quite a bit of time recently thinking about data "versioning" and version control and am actively working on it, see e.g. http://tech.datopian.com/versioning/

You have a nice summary on that page!

Let me illustrate with an example use case. A dataset is released monthly, with each release recorded as a resource in CKAN. So you have a time series, if people download all those monthly resources. However, at one point the methodology used to calculate the data changes and all previous files have to be updated. This is a good opportunity to:

• start providing a change log explaining how and why the methodology changed, which files were changed and how
• increment a version number, ideally still allowing for the download of the (now discontinued) old version

This is based on a real use case.

[rufuspollock]

@augusto-herrmann pretty much what you describe is supported by https://tech.datopian.com/versioning/#ckan-v3 and https://github.com/datopian/ckanext-versions/tree/ckan-2.8

[augusto-herrmann]

That's good to hear! 😄

But what about the f11d specification? What is the best practice? Should each data package contain only a single version or can it be a big package, like a repository of all versions of the dataset (like a git repository is)?

For now, there is a version field in the specification that should be filled with a semantic version numbering system, and that's it. That suggests the former approach, i.e., only one version per data package. But the question is left open: how do we link together those data packages that contain different versions of the same dataset?

[rufuspollock]

I would say the versioning happens outside the Data Package spec ie. that happens in git or CKAN. You could be updating the same data package but making revisions in the versioning system. Does that make sense?

[augusto-herrmann]

That is a possible approach. But it creates a difficulty for data users that would like to automate data collection because it is implementation dependent. Git has one interface, CKAN has another. Having and adopting a data standard is good because you can then reuse code to access the same interface, regardless of what the underlying implementation is.

There should probably be a standard way of listing all versions of the same resource on a data package, listing all versions of a data package, along with the dates, changelog, etc. for each version. It should be easy to do.

Metadata about old versions should be available even after those versions are no longer available – that is one of the FAIR principles of data management. The W3C's Data on the Web Best Practices recommends including data provenance information and data versioning.

The examples there use RDF and the Memento protocol, but in Frictionless it could be as simple as having an optional property "version_history" that would take a list of pairs like {"version_number": "1.1.0", "url": "https://example/data-1.1.0.csv"}. That would link them together and make it easily crawlable.

Other ideas are welcome.