-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reintegrate tracking ID #133
Comments
Ok so in the newest run over at #145 I am able to ingest pretty much whatever I want from the file level like this: import xarray as xr
ds = xr.open_dataset("gs://leap-scratch/data-library/cmip6-pr-copied/8979323652_1/CMIP6.HighResMIP.CMCC.CMCC-CM2-VHR4.highres-future.r1i1p1f1.6hrPlevPt.psl.gn.v20190509.zarr", engine='zarr')
ds.attrs['pangeo_forge_file_data']
|
I have gone through too many cycles of thinking that I have isolated the core/required attributes to extract from the API response. In my latest approach I am literally taking everything I got back from the ESGF API (for dataset and files), and inject it into the attributes: ds.attrs['pangeo_forge_api_responses'] gives something like this now:
{'dataset': {'_timestamp': '2019-06-04T19:25:09.014Z',
'_version_': 1635439145400664064,
'access': ['HTTPServer', 'GridFTP', 'OPENDAP', 'LAS'],
'activity_drs': ['CMIP'],
'activity_id': ['CMIP'],
'cf_standard_name': ['sea_water_salinity'],
'citation_url': ['http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.json'],
'data_node': 'crd-esgf-drc.ec.gc.ca',
'data_specs_version': ['01.00.29'],
'dataset_id_template_': ['%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s'],
'datetime_start': '1850-01-16T12:00:00Z',
'datetime_stop': '2014-12-16T12:00:00Z',
'directory_format_template_': ['%(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s'],
'experiment_id': ['historical'],
'experiment_title': ['all-forcing simulation of the recent past'],
'frequency': ['mon'],
'further_info_url': ['https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.historical.none.r12i1p1f1'],
'grid': ['ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m'],
'grid_label': ['gn'],
'height_bottom': 3.046773,
'height_top': 5624.9517,
'height_units': 'm',
'id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429|crd-esgf-drc.ec.gc.ca',
'index_node': 'esgf-node.llnl.gov',
'instance_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429',
'institution_id': ['CCCma'],
'latest': True,
'master_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn',
'member_id': ['r12i1p1f1'],
'mip_era': ['CMIP6'],
'model_cohort': ['Registered'],
'nominal_resolution': ['100 km'],
'number_of_aggregations': 2,
'number_of_files': 17,
'pid': ['hdl:21.14100/a50b65f2-3554-36b2-9df1-6763bb808840'],
'product': ['model-output'],
'project': ['CMIP6'],
'realm': ['ocean'],
'replica': False,
'retracted': False,
'score': 1.0,
'size': 12682824250,
'source_id': ['CanESM5'],
'source_type': ['AOGCM'],
'sub_experiment_id': ['none'],
'table_id': ['Omon'],
'title': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn',
'type': 'Dataset',
'url': ['http://crd-esgf-drc.ec.gc.ca/thredds/catalog/esgcet/86/CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.xml#CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429|application/xml+thredds|THREDDS',
'http://crd-esgf-drc.ec.gc.ca/las/getUI.do?catid=2D7D52D9ABAF43CDC673C1D95DCBF3BC_ns_CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429|application/las|LAS'],
'variable': ['so'],
'variable_id': ['so'],
'variable_long_name': ['Sea Water Salinity'],
'variable_units': ['0.001'],
'variant_label': ['r12i1p1f1'],
'version': '20190429',
'xlink': ['http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.json|Citation|citation',
'http://hdl.handle.net/hdl:21.14100/a50b65f2-3554-36b2-9df1-6763bb808840|PID|pid']},
'files': [{'_timestamp': '2019-06-04T19:25:09.100Z',
'_version_': 1635439145490841600,
'activity_drs': ['CMIP'],
'activity_id': ['CMIP'],
'cf_standard_name': ['sea_water_salinity'],
'checksum': ['e3dd9be0f144fd6c8e452d5f2f8a4446c5c958eece5ee62219b17fcdf5577d95'],
'checksum_type': ['SHA256'],
'citation_url': ['http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.json'],
'data_node': 'crd-esgf-drc.ec.gc.ca',
'data_specs_version': ['01.00.29'],
'dataset_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429|crd-esgf-drc.ec.gc.ca',
'dataset_id_template_': ['%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s'],
'directory_format_template_': ['%(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s'],
'experiment_id': ['historical'],
'experiment_title': ['all-forcing simulation of the recent past'],
'frequency': ['mon'],
'further_info_url': ['https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.historical.none.r12i1p1f1'],
'grid': ['ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m'],
'grid_label': ['gn'],
'id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_199101-200012.nc|crd-esgf-drc.ec.gc.ca',
'index_node': 'esgf-node.llnl.gov',
'instance_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_199101-200012.nc',
'institution_id': ['CCCma'],
'latest': True,
'master_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.so_Omon_CanESM5_historical_r12i1p1f1_gn_199101-200012.nc',
'member_id': ['r12i1p1f1'],
'mip_era': ['CMIP6'],
'model_cohort': ['Registered'],
'nominal_resolution': ['100 km'],
'pid': ['hdl:21.14100/a50b65f2-3554-36b2-9df1-6763bb808840'],
'product': ['model-output'],
'project': ['CMIP6'],
'realm': ['ocean'],
'replica': False,
'retracted': False,
'score': 1.0,
'size': 770445091,
'source_id': ['CanESM5'],
'source_type': ['AOGCM'],
'sub_experiment_id': ['none'],
'table_id': ['Omon'],
'timestamp': '2019-05-03T08:07:07Z',
'title': 'so_Omon_CanESM5_historical_r12i1p1f1_gn_199101-200012.nc',
'tracking_id': ['hdl:21.14100/981751be-8cc4-4f63-8ac0-ae4f89652788'],
'type': 'File',
'url': ['http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_199101-200012.nc|application/netcdf|HTTPServer',
'gsiftp://crd-esgf-drc.ec.gc.ca:2811//esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_199101-200012.nc|application/gridftp|GridFTP',
'http://crd-esgf-drc.ec.gc.ca/thredds/dodsC/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_199101-200012.nc.html|application/opendap-html|OPENDAP'],
'variable': ['so'],
'variable_id': ['so'],
'variable_long_name': ['Sea Water Salinity'],
'variable_units': ['0.001'],
'variant_label': ['r12i1p1f1'],
'version': '1'},
{'_timestamp': '2019-06-04T19:25:09.191Z',
'_version_': 1635439145586262016,
'activity_drs': ['CMIP'],
'activity_id': ['CMIP'],
'cf_standard_name': ['sea_water_salinity'],
'checksum': ['ed0b8227cdbc518d10d66910a78464876cb59365967c227b454d2881fd490e56'],
'checksum_type': ['SHA256'],
'citation_url': ['http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.json'],
'data_node': 'crd-esgf-drc.ec.gc.ca',
'data_specs_version': ['01.00.29'],
'dataset_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429|crd-esgf-drc.ec.gc.ca',
'dataset_id_template_': ['%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s'],
'directory_format_template_': ['%(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s'],
'experiment_id': ['historical'],
'experiment_title': ['all-forcing simulation of the recent past'],
'frequency': ['mon'],
'further_info_url': ['https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.historical.none.r12i1p1f1'],
'grid': ['ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m'],
'grid_label': ['gn'],
'id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_196101-197012.nc|crd-esgf-drc.ec.gc.ca',
'index_node': 'esgf-node.llnl.gov',
'instance_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_196101-197012.nc',
'institution_id': ['CCCma'],
'latest': True,
'master_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.so_Omon_CanESM5_historical_r12i1p1f1_gn_196101-197012.nc',
'member_id': ['r12i1p1f1'],
'mip_era': ['CMIP6'],
'model_cohort': ['Registered'],
'nominal_resolution': ['100 km'],
'pid': ['hdl:21.14100/a50b65f2-3554-36b2-9df1-6763bb808840'],
'product': ['model-output'],
'project': ['CMIP6'],
'realm': ['ocean'],
'replica': False,
'retracted': False,
'score': 1.0,
'size': 767767513,
'source_id': ['CanESM5'],
'source_type': ['AOGCM'],
'sub_experiment_id': ['none'],
'table_id': ['Omon'],
'timestamp': '2019-05-03T08:04:11Z',
'title': 'so_Omon_CanESM5_historical_r12i1p1f1_gn_196101-197012.nc',
'tracking_id': ['hdl:21.14100/e736e35e-f9dc-4b37-ad6c-fb347cf0d649'],
'type': 'File',
'url': ['http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_196101-197012.nc|application/netcdf|HTTPServer',
'gsiftp://crd-esgf-drc.ec.gc.ca:2811//esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_196101-197012.nc|application/gridftp|GridFTP',
'http://crd-esgf-drc.ec.gc.ca/thredds/dodsC/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_196101-197012.nc.html|application/opendap-html|OPENDAP'],
'variable': ['so'],
'variable_id': ['so'],
'variable_long_name': ['Sea Water Salinity'],
'variable_units': ['0.001'],
'variant_label': ['r12i1p1f1'],
'version': '1'},
{'_timestamp': '2019-06-04T19:25:09.428Z',
'_version_': 1635439145835823104,
'activity_drs': ['CMIP'],
'activity_id': ['CMIP'],
'cf_standard_name': ['sea_water_salinity'],
'checksum': ['1c9cbb98cf421a7f9b7fbe7b3b236afa006df8841538931b309513056b56bfcb'],
'checksum_type': ['SHA256'],
'citation_url': ['http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.json'],
'data_node': 'crd-esgf-drc.ec.gc.ca',
'data_specs_version': ['01.00.29'],
'dataset_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429|crd-esgf-drc.ec.gc.ca',
'dataset_id_template_': ['%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s'],
'directory_format_template_': ['%(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s'],
'experiment_id': ['historical'],
'experiment_title': ['all-forcing simulation of the recent past'],
'frequency': ['mon'],
'further_info_url': ['https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.historical.none.r12i1p1f1'],
'grid': ['ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m'],
'grid_label': ['gn'],
'id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_188101-189012.nc|crd-esgf-drc.ec.gc.ca',
'index_node': 'esgf-node.llnl.gov',
'instance_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_188101-189012.nc',
'institution_id': ['CCCma'],
'latest': True,
'master_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.so_Omon_CanESM5_historical_r12i1p1f1_gn_188101-189012.nc',
'member_id': ['r12i1p1f1'],
'mip_era': ['CMIP6'],
'model_cohort': ['Registered'],
'nominal_resolution': ['100 km'],
'pid': ['hdl:21.14100/a50b65f2-3554-36b2-9df1-6763bb808840'],
'product': ['model-output'],
'project': ['CMIP6'],
'realm': ['ocean'],
'replica': False,
'retracted': False,
'score': 1.0,
'size': 768712941,
'source_id': ['CanESM5'],
'source_type': ['AOGCM'],
'sub_experiment_id': ['none'],
'table_id': ['Omon'],
'timestamp': '2019-05-03T07:56:45Z',
'title': 'so_Omon_CanESM5_historical_r12i1p1f1_gn_188101-189012.nc',
'tracking_id': ['hdl:21.14100/fef027fa-1c02-470e-8fbe-17846573182f'],
'type': 'File',
'url': ['http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_188101-189012.nc|application/netcdf|HTTPServer',
'gsiftp://crd-esgf-drc.ec.gc.ca:2811//esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_188101-189012.nc|application/gridftp|GridFTP',
'http://crd-esgf-drc.ec.gc.ca/thredds/dodsC/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_188101-189012.nc.html|application/opendap-html|OPENDAP'],
'variable': ['so'],
'variable_id': ['so'],
'variable_long_name': ['Sea Water Salinity'],
'variable_units': ['0.001'],
'variant_label': ['r12i1p1f1'],
'version': '1'},
{'_timestamp': '2019-06-04T19:25:09.379Z',
'_version_': 1635439145783394304,
'activity_drs': ['CMIP'],
'activity_id': ['CMIP'],
'cf_standard_name': ['sea_water_salinity'],
'checksum': ['22af932587f79240007e124e277f1fcc86f1dd5a5e61f2159c884afbf7e5fdf9'],
'checksum_type': ['SHA256'],
'citation_url': ['http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.json'],
'data_node': 'crd-esgf-drc.ec.gc.ca',
'data_specs_version': ['01.00.29'],
'dataset_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429|crd-esgf-drc.ec.gc.ca',
'dataset_id_template_': ['%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s'],
'directory_format_template_': ['%(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s'],
'experiment_id': ['historical'],
'experiment_title': ['all-forcing simulation of the recent past'],
'frequency': ['mon'],
'further_info_url': ['https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.historical.none.r12i1p1f1'],
'grid': ['ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m'],
'grid_label': ['gn'],
'id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_190101-191012.nc|crd-esgf-drc.ec.gc.ca',
'index_node': 'esgf-node.llnl.gov',
'instance_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_190101-191012.nc',
'institution_id': ['CCCma'],
'latest': True,
'master_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.so_Omon_CanESM5_historical_r12i1p1f1_gn_190101-191012.nc',
'member_id': ['r12i1p1f1'],
'mip_era': ['CMIP6'],
'model_cohort': ['Registered'],
'nominal_resolution': ['100 km'],
'pid': ['hdl:21.14100/a50b65f2-3554-36b2-9df1-6763bb808840'],
'product': ['model-output'],
'project': ['CMIP6'],
'realm': ['ocean'],
'replica': False,
'retracted': False,
'score': 1.0,
'size': 770016185,
'source_id': ['CanESM5'],
'source_type': ['AOGCM'],
'sub_experiment_id': ['none'],
'table_id': ['Omon'],
'timestamp': '2019-05-03T07:58:35Z',
'title': 'so_Omon_CanESM5_historical_r12i1p1f1_gn_190101-191012.nc',
'tracking_id': ['hdl:21.14100/cb07b0b1-0ec5-477e-949b-dde0c5251408'],
'type': 'File',
'url': ['http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_190101-191012.nc|application/netcdf|HTTPServer',
'gsiftp://crd-esgf-drc.ec.gc.ca:2811//esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_190101-191012.nc|application/gridftp|GridFTP',
'http://crd-esgf-drc.ec.gc.ca/thredds/dodsC/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_190101-191012.nc.html|application/opendap-html|OPENDAP'],
'variable': ['so'],
'variable_id': ['so'],
'variable_long_name': ['Sea Water Salinity'],
'variable_units': ['0.001'],
'variant_label': ['r12i1p1f1'],
'version': '1'},
{'_timestamp': '2019-06-04T19:25:09.293Z',
'_version_': 1635439145693216768,
'activity_drs': ['CMIP'],
'activity_id': ['CMIP'],
'cf_standard_name': ['sea_water_salinity'],
'checksum': ['c4120b7b51eae9318ebecef04bac9d2e18c306dcc4cd751d036fa06552f74dcb'],
'checksum_type': ['SHA256'],
'citation_url': ['http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.json'],
'data_node': 'crd-esgf-drc.ec.gc.ca',
'data_specs_version': ['01.00.29'],
'dataset_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429|crd-esgf-drc.ec.gc.ca',
'dataset_id_template_': ['%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s'],
'directory_format_template_': ['%(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s'],
'experiment_id': ['historical'],
'experiment_title': ['all-forcing simulation of the recent past'],
'frequency': ['mon'],
'further_info_url': ['https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.historical.none.r12i1p1f1'],
'grid': ['ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m'],
'grid_label': ['gn'],
'id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_193101-194012.nc|crd-esgf-drc.ec.gc.ca',
'index_node': 'esgf-node.llnl.gov',
'instance_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_193101-194012.nc',
'institution_id': ['CCCma'],
'latest': True,
'master_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.so_Omon_CanESM5_historical_r12i1p1f1_gn_193101-194012.nc',
'member_id': ['r12i1p1f1'],
'mip_era': ['CMIP6'],
'model_cohort': ['Registered'],
'nominal_resolution': ['100 km'],
'pid': ['hdl:21.14100/a50b65f2-3554-36b2-9df1-6763bb808840'],
'product': ['model-output'],
'project': ['CMIP6'],
'realm': ['ocean'],
'replica': False,
'retracted': False,
'score': 1.0,
'size': 767687293,
'source_id': ['CanESM5'],
'source_type': ['AOGCM'],
'sub_experiment_id': ['none'],
'table_id': ['Omon'],
'timestamp': '2019-05-03T08:01:17Z',
'title': 'so_Omon_CanESM5_historical_r12i1p1f1_gn_193101-194012.nc',
'tracking_id': ['hdl:21.14100/8d514850-727d-42e2-88b4-9a87e0067266'],
'type': 'File',
'url': ['http://crd-esgf-drc.ec.gc.ca/thredds/fileServer/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_193101-194012.nc|application/netcdf|HTTPServer',
'gsiftp://crd-esgf-drc.ec.gc.ca:2811//esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_193101-194012.nc|application/gridftp|GridFTP',
'http://crd-esgf-drc.ec.gc.ca/thredds/dodsC/esgC_dataroot/AR6/CMIP6/CMIP/CCCma/CanESM5/historical/r12i1p1f1/Omon/so/gn/v20190429/so_Omon_CanESM5_historical_r12i1p1f1_gn_193101-194012.nc.html|application/opendap-html|OPENDAP'],
'variable': ['so'],
'variable_id': ['so'],
'variable_long_name': ['Sea Water Salinity'],
'variable_units': ['0.001'],
'variant_label': ['r12i1p1f1'],
'version': '1'},
{'_timestamp': '2019-06-04T19:25:09.072Z',
'_version_': 1635439145461481472,
'activity_drs': ['CMIP'],
'activity_id': ['CMIP'],
'cf_standard_name': ['sea_water_salinity'],
'checksum': ['8f39adbd9c19180ef186004bbd1d96793db147774434b04f07fcf67a2a3a9100'],
'checksum_type': ['SHA256'],
'citation_url': ['http://cera-www.dkrz.de/WDCC/meta/CMIP6/CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.json'],
'data_node': 'crd-esgf-drc.ec.gc.ca',
'data_specs_version': ['01.00.29'],
'dataset_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429|crd-esgf-drc.ec.gc.ca',
'dataset_id_template_': ['%(mip_era)s.%(activity_drs)s.%(institution_id)s.%(source_id)s.%(experiment_id)s.%(member_id)s.%(table_id)s.%(variable_id)s.%(grid_label)s'],
'directory_format_template_': ['%(root)s/%(mip_era)s/%(activity_drs)s/%(institution_id)s/%(source_id)s/%(experiment_id)s/%(member_id)s/%(table_id)s/%(variable_id)s/%(grid_label)s/%(version)s'],
'experiment_id': ['historical'],
'experiment_title': ['all-forcing simulation of the recent past'],
'frequency': ['mon'],
'further_info_url': ['https://furtherinfo.es-doc.org/CMIP6.CCCma.CanESM5.historical.none.r12i1p1f1'],
'grid': ['ORCA1 tripolar grid, 1 deg with refinement to 1/3 deg within 20 degrees of the equator; 361 x 290 longitude/latitude; 45 vertical levels; top grid cell 0-6.19 m'],
'grid_label': ['gn'],
'id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_200101-201012.nc|crd-esgf-drc.ec.gc.ca',
'index_node': 'esgf-node.llnl.gov',
'instance_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.v20190429.so_Omon_CanESM5_historical_r12i1p1f1_gn_200101-201012.nc',
'institution_id': ['CCCma'],
'latest': True,
'master_id': 'CMIP6.CMIP.CCCma.CanESM5.historical.r12i1p1f1.Omon.so.gn.so_Omon_CanESM5_historical_r12i1p1f1_gn_200101-201012.nc',
'member_id': ['r12i1p1f1'],
'mip_era': ['CMIP6'],
'model_cohort': ['Registered'],
'nominal_resolution': ['100 km'],
'pid': ['hdl:21.14100/a50b65f2-3554-36b2-9df1-6763bb808840'],
'product': ['model-output'],
'project': ['CMIP6'],
'realm': ['ocean'],
'replica': False,
'retracted': False,
'score': 1.0,
'size'
Not exactly subtle, but this will leave me with the biggest flexibility to implement tests based on this additional metadata (#99, which should help identify #53, probably avoid #30?) |
Just looking over the original docs for the zarr stores here:
https://pangeo-data.github.io/pangeo-cmip6-cloud/overview.html#zarr-storage-format
We should add the handle_id concatenation to our recipes.
This should also be coordinated with jbusecke/esgf-virtual-zarr-data-access#6 so that the output looks exactly the same to the user!
The text was updated successfully, but these errors were encountered: