Data Package and Data Resource Workflow

[jikaczmarski]

Hi there,

I'm working on a project that has a Data Package standard metadata file (.json). However, what is often the case, is that these individual files should also be available standalone from the data package, and thus I am looking for a programmatic way of exporting the data resources from the data package metadata so that I can package them on their own. I just wanted to reach out here in case this functionality is natively supported before I commit to new tooling for my project.

Here is an example data package:

{
    "profile": "data-package",
    "resources": [
        {
            "name": "resource-0",
            "path": "path-to-resource-0.csv",
            "profile": "tabular-data-resource",
            "schema": {}
        },
        {
            "name": "resource-1",
            "path": "path-to-resource-1.csv",
            "profile": "tabular-data-resource",
            "schema": {}
        }
    ]
}

And here is what I would be hoping to get out for each resource:

{
    "name": "resource-#",
    "path": "path-to-resource-#.csv",
    "profile": "tabular-data-resource",
    "schema": {}
}

To get packaged along with those individual resource files for distribution. Does anyone have any thoughts? Also, let me know if I'm going about this all wrong.

[nichtich]

Maybe I don't fully understand your use case but the transformation from the full datapackage.json to individual resources seems like just a bit of command line magic:

jq -c .resources[] datapackage.json

To further generate individual data resources in subdirectories for each resource, something like this may help:

#!/bin/bash
jq -c .resources[] datapackage.json | \
while read -r res; do
  name=$(echo "$res" | jq -r .name)
  path=$(echo "$res" | jq -r .path)
  mkdir "$name"
  cp "$path" "$name"
  echo "$res" | jq '{profile: "data-package", resources:.}' > "$name/datapackage.json"
done

[gabrielbdornas]

Maybe, as @nichtich, I don't fully understand your use case as well, but the transformation from the full datapackage.json to individual resources could also be done with frictionless python library. I'm using the frictionless 5.18.0 version in the example below:

from frictionless import Package

package = Package('datapackage.json')

for resource_name in package.resource_names:
    resource = package.get_resource(resource_name)
    resource.to_json(f'{resource.name}.json')

It will create a .json file for each resource.

I also created this reprex with the full code.

[jikaczmarski]

Thank you so much for taking the time to answer this. I used a modification of this in one of my larger data packaging functions and it worked flawlessly. Saved me a big headache.