Using IMERG data for observations in MODE

[rvalenzuelar]

Is there any example/tutorial that uses IMERG data for MODE analysis? I have been searching the internet but nothing comes out. I believe it is not trivial because IMERG comes in HDF5 and grid is different to GFS (in my case). I am thinking in the following workflow:

• imerg data (hdf5 file) -> subset with python (save netcdf file)
• make a MET-compliant netCDF file (not sure how)
• use pcp-combine to sum at 6 hour intervals
• apply MODE analysis

Any suggestions?

[JohnHalleyGotway]

I see you have a question about using IMERG data as input to the MODE tool in MET. You're correct that MODE cannot read the HDF5 IMERG data directly, nor can any of the other MET tools. However, MODE, along with the other MET tools, can read gridded data served up via a python embedding script. And that's what I'd recommend you pursue here.

Rather than laying out a prescribed path for processing the data, let me show you what examples already exist and you can decide how to proceed.

First, the MET website does contains some sample Python Scripts to demonstrate python embedding. All of the examples show how to call MET's plot_data_plane tool to read the data via python and plot it. However, there is no example of reading IMERG data there.

Second, a search for "IMERG" in the METplus documentation takes you to this Use Case Page. This is an example of using the METplus wrappers, along with python embedding, to run MET's pcp_combine tool to sum up IMERG data across multiple files. This is pretty much what you're trying to do. It reads HDF5 IMERG data, sums it up with pcp_combine, and writes a NetCDF file that MODE can read.

Here's a link to the actual sum_IMERG_V06_HDF5.py python embedding script that use case calls.

It's up to you if you'd like to get up and running with the METplus wrappers. If so, you can adapt this existing use case for your needs. If not, I'd recommend that you:

• downloads that sum_IMERG_V06_HDF5.py script
• run it manually via python to make sure it runs OK on your data
• run it via MET's plot_data_plane tool to make sure all those connections work
• run it via MET's pcp_combine tool to write the NetCDF output file
• run MODE and read that NetCDF output file

[rvalenzuelar]

Great! METplus was the keyword missing :)
I will report how it goes.

[rvalenzuelar]

Hi @JohnHalleyGotway , reporting some advances here.

I was able to run sum_IMERG_V06_HDF5.py on my machine. I realized that my IMERG files have a RT-H5 extension, instead of the HDF5 extension used by the script, so I had to modify this part.
However, trying to run sum_IMERG_V06_HDF5.py on Docker using the dtcenter/metplus image, returns this error:

Traceback (most recent call last):
  File "/metplus/METplus/parm/use_cases/met_tool_wrapper/PCPCombine/sum_IMERG_V06_HDF5.py", line 5, in <module>
    import h5py
ImportError: No module named h5py

I tried installing h5py with pip within the Docker session but it did not work. Any suggestion?

[georgemccabe]

Hi @rvalenzuelar , using pip to install h5py will install it on a different version of Python than what was used to build the MET executables. You will need to use the MET_PYTHON_EXE environment variable to override the version of Python used to call the Python embedding logic. You can set this to the version of python3 that is in your path to use the version that was used in the pip command. It can either be set directly in the shell:

export MET_PYTHON_EXE=python3

or you can set it only for the call the METplus by passing it in as a user_env_vars configuration override argument to run_metplus.py:

run_metplus.py ... user_env_vars.MET_PYTHON_EXE=python3

The version of Python used must also have the required packages needed for Python Embedding, so you may also need to install netcdf4 and xarray as well.

Let me know if you have any issues getting this set up.

[rvalenzuelar]

Thanks @georgemccabe, this is what I did (following your directions):

run pip3 install h5py
run python3 sum_IMERG_V06_HDF5.py [path_to_imerg_files] precipitationCal [accumulation_ending_time] [accumulation_hours]
and it worked correctly.

Then, I edited the config file for PCPcombine, using MET_PYTHON_EXE = /usr/bin/python3 at the end of the file.

This workflow worked fine.

Thanks for your assistance!

[georgemccabe]

@rvalenzuelar, that is great to hear! Thanks for letting us know.