Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Pyaro EBAS reader for NILU PMF data, as well as general use #32

Closed
dulte opened this issue Apr 9, 2024 · 2 comments · Fixed by #35
Closed

Add Pyaro EBAS reader for NILU PMF data, as well as general use #32

dulte opened this issue Apr 9, 2024 · 2 comments · Fixed by #35
Assignees
Milestone

Comments

@dulte
Copy link
Collaborator

dulte commented Apr 9, 2024

The NILU PMF data that I want to read is found here: https://secondary-data-archive.nilu.no/ebas/gen.h8ds-8596/EIMPs_winter2017-2018_data.zip

The files are under the "EIMPs_winter_2017_2018_ECOC_Levo" folder. The other folder can be read by PR #31, but with a new EBAS reader, that reader might become redundant

@jgriesfeller jgriesfeller added this to the m2024-05 milestone Apr 9, 2024
@jgriesfeller
Copy link
Member

jgriesfeller commented Apr 10, 2024

Note:
In principle it should have been possible to read these files with pyaerocom using the read_file method of the ReadEbas class. Unfortunately the format is slighly different to what we usually get:

from pyaerocom.io.read_ebas import ReadEbas


def main():
    filename = "/home/jang/tmp/EIMPs_winter2017-2018_data/EIMPs_winter_2017_2018_ECOC_Levo/AT0002R.20171130000000.20180809104500.high_vol_sampler.oc_ec_tc.pm25.3mo.3d.AT05L_hvs_02.AT05L_TU_sunset_001.lev2.nas"
    data = ReadEbas()
    data_out = data.read_file(filename, vars_to_retrieve="concoc")
    assert data_out


if __name__ == "__main__":
    main()

ends with

  File "/home/jang/micromamba/envs/pyadev-applied/lib/python3.11/site-packages/pyaerocom/io/read_ebas.py", line 1318, in find_var_cols
    col = self._find_best_data_column(cols, self.get_ebas_var(var), file)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jang/micromamba/envs/pyadev-applied/lib/python3.11/site-packages/pyaerocom/io/read_ebas.py", line 951, in _find_best_data_column
    raise EbasFileError(
pyaerocom.exceptions.EbasFileError: Cannot infer data statistics for data column name: organic_carbon, unit: ug C/m3, is_var: True, is_flag: False, flag_col: 11, detection_limit: 0.08 ug C/m3. Neither column nor file meta specifications include information about data statistics

We might want to discuss to which extend we reimplement the EBAS reader in pyaro-readers, e.g. should pyaro deliver the data in terms of pyaerocom variables or in terms of EBAS variables. This is unfortunately not a 1:1 relationship

@jgriesfeller
Copy link
Member

Summary from discussion @ the pyaro-dev channel: implement seperate reader here. The original one in pyaerocom will not be working anymore as soon as the ACTRIS EBAS reader is working due to no new data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants