Help with Python embedding for MTD

[anwe5]

Hi,

I’m working on using MTD on my data, which has grid mapping type "rotated_pole". Since this grid type isn't supported by MET, I’m using Python embedding as a workaround. However, I’m encountering some challenges with using Python embedding and setting up the config file. I’m not sure if I’m missing any key steps or configurations when using Python embedding with MTD.

I use a python environment where I set the following variables:

export MET_BASE=/home/b/b382378/MET/MET_releases/11.1.0/share/met/
export PATH=$PATH:/home/b/b382378/MET/MET_releases/11.1.0/bin
export PYTHONPATH=/home/b/b382378/MET/MET_releases/11.1.0/share/met/python
export MET_PYTHON_EXE=/sw/spack-levante/python-3.10.10-p24a46/bin/python3

Here, you can find the config file, Python embedding script, a description of the grid, and the MTD_test.log file:
files.zip

When I run MTD with mtd -single /scratch/b/b382378/test_data_MTD/hourly_data_* -config /home/b/b382378/MET/test_MTD/MTDConfig_test -outdir /scratch/b/b382378/MTD_out -log /home/b/b382378/MET/test_MTD/MTD_test.log -v 1 I get the following error message:

DEBUG 1: Start mtd by b382378(275502) at 2024-08-19 14:48:12Z  cmd: mtd -single /scratch/b/b382378/test_data_MTD/hourly_data_00.nc /scratch/b/b382378/test_data_MTD/hourly_data_01.nc /scratch/b/b382378/test_data_MTD/hourly_data_02.nc /scratch/b/b382378/test_data_MTD/hourly_data_03.nc /scratch/b/b382378/test_data_MTD/hourly_data_04.nc /scratch/b/b382378/test_data_MTD/hourly_data_05.nc /scratch/b/b382378/test_data_MTD/hourly_data_06.nc /scratch/b/b382378/test_data_MTD/hourly_data_07.nc /scratch/b/b382378/test_data_MTD/hourly_data_08.nc /scratch/b/b382378/test_data_MTD/hourly_data_09.nc /scratch/b/b382378/test_data_MTD/hourly_data_10.nc /scratch/b/b382378/test_data_MTD/hourly_data_11.nc /scratch/b/b382378/test_data_MTD/hourly_data_12.nc /scratch/b/b382378/test_data_MTD/hourly_data_13.nc /scratch/b/b382378/test_data_MTD/hourly_data_14.nc /scratch/b/b382378/test_data_MTD/hourly_data_15.nc /scratch/b/b382378/test_data_MTD/hourly_data_16.nc /scratch/b/b382378/test_data_MTD/hourly_data_17.nc /scratch/b/b382378/test_data_MTD/hourly_data_18.nc /scratch/b/b382378/test_data_MTD/hourly_data_19.nc /scratch/b/b382378/test_data_MTD/hourly_data_20.nc /scratch/b/b382378/test_data_MTD/hourly_data_21.nc /scratch/b/b382378/test_data_MTD/hourly_data_22.nc /scratch/b/b382378/test_data_MTD/hourly_data_23.nc -config /home/b/b382378/MET/test_MTD/MTDConfig_test -outdir /scratch/b/b382378/MTD_out -log /home/b/b382378/MET/test_MTD/MTD_test.log -v 1
Traceback (most recent call last):
  File "/home/b/b382378/MET/MET_releases/11.1.0/share/met//python/pyembed/write_tmp_dataplane.py", line 19, in <module>
    from met.dataplane import dataplane
  File "/home/b/b382378/MET/MET_releases/11.1.0/share/met/python/met/dataplane.py", line 3, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
ERROR  :
ERROR  : tmp_nc_dataplane() -> command "/sw/spack-levante/python-3.10.10-p24a46/bin/python3 /home/b/b382378/MET/MET_releases/11.1.0/share/met//python/pyembed/write_tmp_dataplane.py /tmp/tmp_met_nc_1475718_0 APCP" failed ... status = 256
ERROR  :

I tried running dataplane.py separately, and it works without any error messages (including loading numpy, xarray, etc.). I also completed the online tutorial for MTD, and it worked without any issues on the tutorial data. Therefore, I suspect the error might be related to my use of Python embedding.

Could you provide some guidance on how to set up the Python embedding?

If you need any additional information or files, please let me know!

Thanks in advance for your help!

Best regards,
Annabell

[jprestop]

Hi Annabell (@anwe5).

I see that the problem is that numpy cannot be found:

import numpy as np
ModuleNotFoundError: No module named 'numpy'

when using /sw/spack-levante/python-3.10.10-p24a46/bin/python3.

Can you please run the following commands?

/sw/spack-levante/python-3.10.10-p24a46/bin/python3
import numpy

to determine if numpy loads without a problem.

If numpy does load, please try removing the following line from your python_embedding.py script and rerun:

#!/usr/bin/env python                                                                                                                                                                                                                                                                                      

Please let us know how it goes.

Regarding guidance on setting up the Python embedding, please take a look at the documentation in our MET User's Guide. Hopefully, this will answer any questions you have. But, please let us know if you do have any further questions after reading through our documentation: 37. Appendix F Python Embedding

[anwe5]

Results of the first test (with setting MET_PYTHON_EXE):

plot_data_plane test.ps 'name="/home/b/b382378/MET/test_MTD/embedding_new.py /scratch/b/b382378/test_data_MTD/hourly_data_00.nc";' -v 5
DEBUG 1: Start plot_data_plane by b382378(275502) at 2024-08-29 09:28:12Z  cmd: plot_data_plane test.ps name="/home/b/b382378/MET/test_MTD/embedding_new.py /scratch/b/b382378/test_data_MTD/hourly_data_00.nc"; -v 5

*** Model Evaluation Tools (METV11.1.1) ***

Usage: plot_data_plane
        input_filename
        output_filename
        field_string
        [-color_table color_table_name]
        [-plot_range min max]
        [-title title_string]
        [-log file]
        [-v level]

        where   "input_filename" is the name of a gridded data file to be plotted (required).
                "output_filename" is the name of the output PostScript file to be written (required).
                "field_string" defines the data to be plotted from the input file (required).
                "-color_table color_table_name" overrides the default color table ("colortables/met_default.ctable") (optional).
                "-plot_range min max" defines the range of the data to be plotted (optional).
                "-title title_string" specifies the plot title string (optional).
                "-log file" outputs log messages to the specified file (optional).
                "-v level" overrides the default level of logging (5) (optional).

For the first test, it seems that the command line is not correct. I tried to fix it, but it didn't work. I also tried the example code from Listing 37.8 of the User Guide Appendix F :

plot_data_plane PYTHON_NUMPY fcst.ps 'name="scripts/python/examples/read_ascii_numpy.py data/python/fcst.txt FCST";' -title "Python enabled plot_data_plane"
DEBUG 1: Start plot_data_plane by b382378(275502) at 2024-08-29 09:24:29Z  cmd: plot_data_plane PYTHON_NUMPY fcst.ps name="scripts/python/examples/read_ascii_numpy.py data/python/fcst.txt FCST"; -title Python enabled plot_data_plane
DEBUG 1: Opening data file: PYTHON_NUMPY
Python Script:  '/home/b/b382378/MET/MET_releases/11.1.0/share/met//python/pyembed/write_tmp_dataplane.py'
User Command:   'scripts/python/examples/read_ascii_numpy data/python/fcst.txt FCST'
Temporary File: '/tmp/tmp_met_nc_2115123_0'
 PYTHON: Python Script: 'scripts/python/examples/read_ascii_numpy'
 PYTHON: Input File:    'data/python/fcst.txt'
 PYTHON: Data Name:     'FCST'
 PYTHON: Data Shape:    (129, 185)
 PYTHON: Data Type:     dtype('float64')
 PYTHON: Attributes:    {'valid': '20050807_120000', 'init': '20050807_000000', 'lead': '120000', 'accum': '120000', 'name': 'FCST', 'long_name': 'FCST_word', 'level': 'Surface', 'units': 'None', 'grid': {'type': 'Lambert Conformal', 'hemisphere': 'N', 'name': 'FooGrid', 'scale_lat_1': 25.0, 'scale_lat_2': 25.0, 'lat_pin': 12.19, 'lon_pin': -135.459, 'x_pin': 0.0, 'y_pin': 0.0, 'lon_orient': -95.0, 'd_km': 40.635, 'r_km': 6371.2, 'nx': 185, 'ny': 129}}
Traceback (most recent call last):
  File "/home/b/b382378/MET/MET_releases/11.1.0/share/met/python/met/dataplane.py", line 4, in <module>
    import netCDF4 as nc
  File "/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/site-packages/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
ImportError: /sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/site-packages/netCDF4/../../../libnetcdf.so.19: undefined symbol: H5Pset_fapl_ros3
WARNING:
WARNING: tmp_nc_dataplane() -> an error occurred importing module "met.dataplane"
WARNING:
ERROR  :
ERROR  : plot_data_plane -> trouble getting field "name="scripts/python/examples/read_ascii_numpy.py data/python/fcst.txt FCST";" from file "PYTHON_NUMPY"
ERROR  :

Result of the second test (without setting MET_PYTHON_EXE):
Same output if I use the command you suggested.

For the example from Appendix F, I get:

plot_data_plane PYTHON_NUMPY fcst.ps \
> 'name="scripts/python/examples/read_ascii_numpy.py data/python/fcst.txt FCST";' \
> -title "Python enabled plot_data_plane"
DEBUG 1: Start plot_data_plane by b382378(275502) at 2024-08-29 10:01:00Z  cmd: plot_data_plane PYTHON_NUMPY fcst.ps name="scripts/python/examples/read_ascii_numpy.py data/python/fcst.txt FCST"; -title Python enabled plot_data_plane
DEBUG 1: Opening data file: PYTHON_NUMPY
Traceback (most recent call last):
  File "/home/b/b382378/MET/MET_releases/11.1.0/share/met//python/met/dataplane.py", line 4, in <module>
    import netCDF4 as nc
  File "/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/site-packages/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
ImportError: /sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/site-packages/netCDF4/../../../libnetcdf.so.19: undefined symbol: H5Pset_fapl_ros3
WARNING:
WARNING: straight_python_dataplane() -> an error occurred importing module "scripts/python/examples/read_ascii_numpy"
WARNING:
ERROR  :
ERROR  : plot_data_plane -> trouble getting field "name="scripts/python/examples/read_ascii_numpy.py data/python/fcst.txt FCST";" from file "PYTHON_NUMPY"
ERROR  :

About the netcdf Python module:
I forgot to mention that on Levante, the netcdf-c and netcdf-cxx libraries are not installed in the same directory. I used "stow" to link both libraries into one directory before installing MET, as recommended by the DKRZ helpdesk.

However, when I tested the data from the MET tutorial, everything worked fine. If the issue is related to the linkage, I would expect the tutorial to have failed as well, right?

[DanielAdriaansen]

I'm so sorry. The command was malformed (as you suspected), my apologies!

Can you try again with this command:
plot_data_plane PYTHON_NUMPY test.ps 'name="/home/b/b382378/MET/test_MTD/embedding_new.py /scratch/b/b382378/test_data_MTD/hourly_data_00.nc";' -v 5

Also, one of our engineers @hsoh-u suggested doing the following:

1. Check the values of these two environment variables:

echo $MET_PYTHON_BIN_EXE
echo $MET_PYTHON_EXE

2. If either or both return a value, then issue these commands for the matching environment variable:

$MET_PYTHON_BIN_EXE -c " import netCDF4"
$MET_PYTHON_EXE -c " import netCDF4"

If those two commands do not return an error, then it indicates netCDF4 is available in both Python installations (MET_PYTHON_BIN_EXE is the Python MET was compiled against, and MET_PYTHON_EXE is the Python you are using for Python embedding).

Let us know how it goes and again, my profuse apologies for the command line error.

[anwe5]

Thank you for providing a new version of the command!

Should I set MET_PYTHON_BIN_EXE and MET_PYTHON_EXE within the Python environment I created for MET?

Here is the code from the Python environment:

export PATH=$PATH:/home/b/b382378/MET/MET_releases/11.1.0/bin
export MET_PYTHON_EXE=/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/bin/python3
export MET_PYTHON_BIN_EXE=/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/bin/python3

When I run "echo" + the variables outside of the environment, it returns an empty line. When I run it within the Python environment, it returns /sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/bin/python3 for both variables, which seems reasonable.

Should these variables be set differently from each other? Or do I even need to set them in the environment to run MET tools?

Importing netCDF4 wokrs without any errors within the environment. However, it does not work outside the Python environment, as expected:

[b382378@levante3 ~]$ $MET_PYTHON_EXE -c "import netCDF4"
-bash: -c: command not found
[b382378@levante3 ~]$ $MET_PYTHON_BIN_EXE -c "import netCDF4"
-bash: -c: command not found
[b382378@levante3 ~]$ $MET_PYTHON_BIN_EXE "import netCDF4"
-bash: import netCDF4: command not found

Here is the output of the test with plot_data_plane:

plot_data_plane PYTHON_NUMPY test.ps 'name="/home/b/b382378/MET/test_MTD/embedding_new.py /scratch/b/b382378/test_data_MTD/hourly_data_00.nc";' -v 5
DEBUG 1: Start plot_data_plane by b382378(275502) at 2024-08-30 12:34:11Z  cmd: plot_data_plane PYTHON_NUMPY test.ps name="/home/b/b382378/MET/test_MTD/embedding_new.py /scratch/b/b382378/test_data_MTD/hourly_data_00.nc"; -v 5
DEBUG 1: Opening data file: PYTHON_NUMPY
DEBUG 4: Met2dDataFileFactory::new_met_2d_data_file() -> created new Met2dDataFile object of type "FileType_Python_Numpy".
DEBUG 4: VarInfoFactory::new_var_info() -> created new VarInfo object of type "FileType_Python_Numpy".
DEBUG 3: Calling /sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/bin/python3 to run user's python script (/home/b/b382378/MET/test_MTD/embedding_new).
DEBUG 4: Writing temporary Python dataplane file:
DEBUG 4:        /sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/bin/python3 /home/b/b382378/MET/MET_releases/11.1.0/share/met//python/pyembed/write_tmp_dataplane.py /tmp/tmp_met_nc_388514_0 /home/b/b382378/MET/test_MTD/embedding_new /scratch/b/b382378/test_data_MTD/hourly_data_00.nc
Python Script:  '/home/b/b382378/MET/MET_releases/11.1.0/share/met//python/pyembed/write_tmp_dataplane.py'
User Command:   '/home/b/b382378/MET/test_MTD/embedding_new /scratch/b/b382378/test_data_MTD/hourly_data_00.nc'
Temporary File: '/tmp/tmp_met_nc_388514_0'
DEBUG 3: Initializing MET compile time python instance: /sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/bin/python3
DEBUG 4: Reading temporary Python dataplane file: /tmp/tmp_met_nc_388514_0
Traceback (most recent call last):
  File "/home/b/b382378/MET/MET_releases/11.1.0/share/met//python/met/dataplane.py", line 4, in <module>
    import netCDF4 as nc
  File "/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/site-packages/netCDF4/__init__.py", line 3, in <module>
    from ._netCDF4 import *
ImportError: /sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/site-packages/netCDF4/../../../libnetcdf.so.19: undefined symbol: H5Pset_fapl_ros3
WARNING:
WARNING: tmp_nc_dataplane() -> an error occurred importing module "met.dataplane"
WARNING:
ERROR  :
ERROR  : plot_data_plane -> trouble getting field "name="/home/b/b382378/MET/test_MTD/embedding_new.py /scratch/b/b382378/test_data_MTD/hourly_data_00.nc";" from file "PYTHON_NUMPY"
ERROR  :

I encounter the same error when using PYTHON_XARRAY instead of PYTHON_NUMPY, and when running it outside of the environment.

If I comment out import numpy as np and import xarray as xr in the embedding_new.py script, I receive the following error:

NameError: name 'xr' is not defined
ERROR  :
ERROR  : tmp_nc_dataplane() -> command "/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/bin/python3 /home/b/b382378/MET/MET_releases/11.1.0/share/met//python/pyembed/write_tmp_dataplane.py /tmp/tmp_met_nc_437156_0 /home/b/b382378/MET/test_MTD/embedding_new /scratch/b/b382378/test_data_MTD/hourly_data_00.nc" failed ... status = 256
ERROR  :

Thanks for your help!

[anwe5]

Hi,

I resolved the issue with the Python version. I realized I wasn't using the correct versions of Python and NetCDF for the MET installation. After correcting this, the 'plot_data_plane' tool is now working.

Thank you very much for your help on this issue!

[DanielAdriaansen]

Great news @anwe5! We were really scratching our heads over here. If there's anything else don't hesitate to reach out.