METplus use case errors METdbLoad 5.1.13.1, Marine and Cryosphere 5.2.5.2 and 5.2.5.6

[tmalcala]

I am having multiple issues when running some METplus use cases. When running the METdbLoad 5.1.13.1 use case I am getting a pandas error stating

pandas.errors.ParserError: Too many columns specified: expected 125 and found 24

When running two of the Marine and Cryosphere use cases, 5.2.5.2 GridStat: Python Embedding to read and process ice cover and 5.3.5.6 PlotDataPlane: Python Embedding of tripolar coordinate file, I get the same error stating

/sw/afw/python/anaconda3/base/envs/afw-py39-202203/lib/python3.9/site-packages/pyresample/kd_tree.py:384: UserWarning: Possible more than 8 neighbours within 50000 m for some data points warnings.warn(('Possible more than %s neighbours ' ERROR : ERROR : Python3_Numpy::set(PyObject *)-> failed to get buffer

We are using newer versions of the METplus Python packages so I am not sure if it may be an issue with how the sample data is formatted that newer versions now don't accept or anything like that. I have attached the full logs for each of the errors as well as a list of our Python packages installed with their versions.

metpluspandas.log
python_packages.txt
numpy_buffer.log

[JohnHalleyGotway]

@tmalcala I see that you're having some python issues here!

My colleague @georgemccabe commented that this is likely due to the METdataio pandas version issues we have been discussing recently. @venitahagerty has found that there is a bug in pandas 1.4.2 that is causing this issue. The bug does not exist using pandas 1.2.3 but it appears to have been reintroduced since that version.'

@venitahagerty can you offer any suggestions for @tmalcala on how best to proceed?

[venitahagerty]

With the change to python 3.8, we have recently discovered a problem reading some data with pandas 1.4. The develop branch for METdbLoad currently requires pandas 1.2.3. We are looking at how to support newer versions in the future.

[tmalcala]

Alright, we aren't admins for the system it is on so I will see if it is possible to use an older version for now without disrupting anything else. Any recommendations for the "failed to get buffer" error?

[georgemccabe]

Hi @tmalcala, regarding the "failed to get buffer" error:

I ran the ice cover use case locally and while I did see the UserWarning you see, I did not get the same error that you see. I would suggest running the Python Embedding script outside of METplus to see if any errors are output that may suggest why it failed. You command would look something like this:

python3 /lustre/cyclone/nwp607/scratch/alcalat/metplus/metplus-TADS-15486/parm/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsOSTIA_iceCover/read_ice_data.py ./metplus_sample/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsOSTIA_iceCover/20210305_rtofs_glo_2ds_n024_ice.nc ./metplus_sample/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsOSTIA_iceCover/2021030512_UKMO_L4.nc north fcst

Make sure that the version of python3 used to run the script is the same version that is used to execute Python Embedding through METplus.

[JohnHalleyGotway]

@tmalcala I work with @georgemccabe on providing METplus support. So it looks like that python script ran when run directly.

A good potential next step would be running it through MET's plot_data_plane tool. That reads gridded input data (either from a file or via python embedding) and just plots the results. That's a good test to make sure the "machinery" is working.

Please try running:

export MET_PYTHON_EXE=`which python3`
${MET_BIN}/plot_data_plane PYTHON_NUMPY plot.ps 'name="/lustre/cyclone/nwp607/scratch/alcalat/metplus/metplus-TADS-15486/parm/use_cases/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsOSTIA_iceCover/read_ice_data.py ./metplus_sample/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsOSTIA_iceCover/20210305_rtofs_glo_2ds_n024_ice.nc ./metplus_sample/model_applications/marine_and_cryosphere/GridStat_fcstRTOFS_obsOSTIA_iceCover/2021030512_UKMO_L4.nc north fcst";'

... where you'd replace MET_BIN with the path to where the MET executables are installed.

The question is whether this runs without error? Will it succeed or will it error out with the "failed to get buffer" error?

[tmalcala]

Got a similar result, just the warning, no error

[tmalcala]

Any guesses on this?

[JohnHalleyGotway]

Great, glad that the plot_data_plane tool runs on the command line without error.

Now that you've set the $MET_PYTHON_EXE environment variable, have you tried re-running that use case with it set? If not, please do so and let us know the result.

If the use case still fails with $MET_PYTHON_EXE set, please:

• Tell us the version of MET you're running.
• Send us the config.log file from the top-level MET directory.

[tmalcala]

That worked! All of the METplus use cases now pass. Thank you!