Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backfill newly injected attributes #155

Open
jbusecke opened this issue May 8, 2024 · 0 comments
Open

Backfill newly injected attributes #155

jbusecke opened this issue May 8, 2024 · 0 comments
Labels

Comments

@jbusecke
Copy link
Collaborator

jbusecke commented May 8, 2024

In the future we might refactor our tests to depend on attributes injected by the recipe ( like here).
These attributes will only be available on very recent stores.

We have the following options:

  • Strictly dismiss anything that does not contain the current set of required metadata (that would be too harsh)
  • Once we settle on a set of metadata to include we should see if it is possible to have a 'backfill script' that iterates through all stores, and uses existing metadata to fill dataset_level meta data by querying the ESGF API -- I don't think we will be able to deduce file level data since those old stores do not retain enough info -- and overwriting the zarr stores. (they should clearly be marked by some additional attribute like pangeo_forge_metadata_backfilled or similar.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant