Skip to content

2.0.0 (May 15, 2023)
Compare
Choose a tag to compare
@FjodorvRijsselberg FjodorvRijsselberg released this 15 May 14:06
· 229 commits to main since this release
v2.0.0
511a0e7
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Dynaconf configuration

The settings for the different data providers and target dataverse instances have been moved to settings tomls in the configuration directory. A entry ingestion workflow can now have a parameter to specify which settings dictionary it will use. By using this setup all workflows for specific dataverses or subverses have been removed.

min.io

The data that is ingested by the workflows in the orchestrator is now expected to be in a bucket in a min.io object store. Local data ingestion is no longer possible. The setup for the object store needs to be added to the .secrets.toml in the configuration directory.

Universal Dataverse2Dataverse ingestion workflow

All dataverse to dataverse ingestion is now done using the same workflow. Any specific differences between the source metadata that need to be refined are handled in the metadata-refiner service.

Minor changes

  • Dockerfile update to allow for the easy addition of new poetry packages.
  • Added jmespath for querying fields from JSON metadata.
  • Mapping file has been updated for use with the new dataverse-mapper.
  • CBS ingestion workflow now includes an email sanitation task.
  • Added metadata refinement task that is used in d2d ingestion workflow.
  • Refactored xml2json task to work with metadata fetched from minio.
  • Metadata fetcher service now no longer needs a Dataverse source API key.
Assets 2
Loading