Feature-relative use case with WRF data?

[nleonardo87]

Hi,
I am trying to run the feature-relative use-case with 4-km WRF data, processing a 2D variable (PBL height). The variable is indeed compatible with MET, as I can run plot_data_plane.

However, I get the following error when it's running series_analysis:

• ERROR : MetNcFile::data(NcVar *, const LongArray &, DataPlane &) const -> needed 2 arguments for variable PBLH, got 3

This seems to suggest that it thinks the variable is 3D instead of 2D? Technically, it is 3D (Time, south_north, west_east), though the first dimension only has 1 element (one time per wrfout file). Is there a way around this?

I can copy/paste the config file and full error logs, but they are quite long. Thank you.
-Nick

[JohnHalleyGotway]

Hello Nick. I see you're having trouble running the METplus feature relative use case on WRF output. Rather than copy/pasting the full error logs and your METplus configuration files into the message, please just drag/drop them as an attachment. If GitHub complains about the filename suffixes, please just store them in a tar.gz file. For example:

tar -cvzf METplus_data.tar.gz *.log *.conf

This command creates are tarred/zipped file containing all the files that match *.log or *.conf. GitHub will allow you to attach that tar.gz file to this discussion.

@georgemccabe is the best person to actually address this issue, and he'll be back in the office tomorrow.

I do believe that for the feature relative use case, METplus calls the regrid_data_plane tool on the input WRF files to cut out tiles centered on the feature (i.e. hurricane center location). Those calls to regrid_data_plane read data from the WRFOUT file and will need 'level = "(0,*,*)";' to do so. However, the next step in the use case calls the series_analysis tool using those tiles as input. But that call to series_analysis needs 'level = "(*,*)";' since the MET NetCDF output currently does NOT include the time dimension.

Hopefully @georgemccabe can sort through the details and recommend how to tweak your METplus configuration to get these steps working.

[georgemccabe]

Hi Nick,

I believe @JohnHalleyGotway is on the right track in regards to this error. In the existing feature relative examples, the field info can be defined the same for ExtractTiles and SeriesAnalysis wrappers, however in your case you will need to define the field info for each tool separately.

Instead of defining:

BOTH_VAR1_NAME = PBLH
BOTH_VAR1_LEVELS = "(0,*,*)"

you should try setting:

BOTH_EXTRACT_TILES_VAR1_NAME = PBLH
BOTH_EXTRACT_TILES_VAR1_LEVELS = "(0,*,*)"

BOTH_SERIES_ANALYSIS_VAR1_NAME = PBLH
BOTH_SERIES_ANALYSIS_VAR1_LEVELS = "(*,*)"

This configuration will read a 2D slab of data from the 3D input field into regrid_data_plane via ExtractTiles wrapper and write a 2D field that will be read into SeriesAnalysis. The exact field configuration may slightly differ from the above example since I can't see what you currently have configured. If you are still unable to get it to work properly, please send the information that John requested so I can take a closer look. Thanks.

[nleonardo87]

Thank you. I'll try that first chance I get. Here is my config file and the error log from my latest attempt (before adding the changes), in case there's anything else.
METplus_data.tar.gz

[georgemccabe]

Hi @nleonardo87, I am just following up on this issue. Were you able to get the results you expect with the suggested changes?

[nleonardo87]

Sorry for the delay. It's been a very chaotic couple of weeks. I replaced the section you mentioned, but it did not work. The first error it returns seems to involve it being unable to find the wrfout file... even though it had gotten past this part in my last attempt and the file/path it claims is invalid is definitely there.
.
.
.
OUTPUT:
DEBUG 1: Reading data file: /glade/u/home/nleonardo/ELSA/20210709/WRFgfs4_20210709_1200_000.nc
ERROR :
ERROR : process_data_file() -> "/glade/u/home/nleonardo/ELSA/20210709/WRFgfs4_20210709_1200_000.nc" not a valid data file
ERROR :
.
.
.

Reincluding the "BOTH_VAR1_..." flags at least gets to the point where it reads the file, but then it says.
.
.
.
DEBUG 2: Processing series entry 1 of 1: PBLH(,) versus PBLH(,)
WARNING:
WARNING: get_series_entry() -> No match found for PBLH(,) in file: /glade/u/home/nleonardo/METplus_Tutorial/ATTEMPT001/output/extract_tiles/20210709_12/AL052021/FCST_TILE_F000_Wgfs4_PBLH_20210709_1200_000.nc
WARNING:
ERROR :
ERROR : DataPlane::set_constant(double) -> no data buffer allocated!
ERROR :
.
.
.

Incidentally, I work on a Windows laptop and have been unable to ssh to Cheyenne directly. I got around this by ssh-ing to my University's cluster and from there to Cheyenne. However, we recently updated the cluster. After removing the old keys on both ends and so forth, I'm still able to ssh from my cluster to Cheyenne, but I am unable to transfer files from Cheyenne to my cluster. I get this error now.

Unable to negotiate with ... port 22: no matching key exchange method found. Their offer: ...
lost connection

It doesn't seem to be a problem with my university's cluster. What can I do to get around this on Cheyenne? Thank you.

[georgemccabe]

Hi @nleonardo87, it looks like you had this set in your original conf file:

BOTH_VAR1_OPTIONS = file_type = NETCDF_PINT

so you will need to also set that for the ExtractTiles specific variable:

BOTH_EXTRACT_TILES_VAR1_OPTIONS = file_type = NETCDF_PINT

That should tell the tool that the file type is a WRF out file and handle it accordingly. There is a chance this will not work, as I don't think anyone has had to pass in extra options to define fields for ExtractTiles. If not, we will have to enhance the wrapper to allow it. Let me know what you find.

Regarding file transfers from Cheyenne, I found this resource about data transfers that may be helpful:
https://www2.cisl.ucar.edu/resources/storage-and-file-systems/data-transfers-and-sharing

If not, I would recommend contacting CISL Help Desk for support.

[nleonardo87]

Thank you. I added the flag you mentioned, such that my config file looks like:
_.
.
.
BOTH_VAR1_NAME = PBLH
BOTH_VAR1_LEVELS = "(0,,)"
BOTH_VAR1_OPTIONS = file_type = NETCDF_PINT

BOTH_EXTRACT_TILES_VAR1_NAME = PBLH
BOTH_EXTRACT_TILES_VAR1_LEVELS = "(0,,)"
BOTH_EXTRACT_TILES_VAR1_OPTIONS = file_type = NETCDF_PINT

BOTH_SERIES_ANALYSIS_VAR1_NAME = PBLH
BOTH_SERIES_ANALYSIS_VAR1_LEVELS = "(,)"
.
.
._

I still get the error:
.
.
.
DEBUG 1: The "fcst.field" and "obs.field" configuration entries and the "-fcst" and "-obs" command line options all have length one.
DEBUG 2: Processing series entry 1 of 1: PBLH(,) versus PBLH(,)
WARNING:
WARNING: get_series_entry() -> No match found for PBLH(,) in file: /glade/u/home/nleonardo/METplus_Tutorial/ATTEMPT001/output/extract_tiles/20210709_12/AL052021/FCST_TILE_F000_Wgfs4_PBLH_20210709_1200_000.nc
WARNING:
ERROR :
ERROR : DataPlane::set_constant(double) -> no data buffer allocated!
ERROR :
.
.
.

[georgemccabe]

Could you check the file that is being read to see what fields are in it?

/glade/u/home/nleonardo/METplus_Tutorial/ATTEMPT001/output/extract_tiles/20210709_12/AL052021/FCST_TILE_F000_Wgfs4_PBLH_20210709_1200_000.nc

There should be a 2D field named PBLH in that file that was generated by regrid_data_plane via ExtractTiles wrapper.

It is also odd that SeriesAnalysis wrapper is only finding 1 file for each fcst and obs, as there should be at least 2 for each. If you rerun with LOG_LEVEL = DEBUG set in your METplus config file you will get more output that may provide an idea as to why this is happening.

I also noticed that you do not set TC_PAIRS_INIT_BEG or TC_PAIRS_INIT_END, which is used to filter out data to use. When these are not set, the value of INIT_BEG and INIT_END are used. In this case, they are set to the same time, so you may be filtering out a lot of data that you want to use.

If you are still having issues I can log into Cheyenne and run the use case to take a closer look at what is going on.

[nleonardo87]

It's strange... There is 2D (40-by-40) field named PBLH in that file, but it's empty. This variable is definitely not empty in the original wrfout files.

I added the LOG_LEVEL = DEBUG flag, but it's not clear to me why it's failing. I still get:
...
10/14 13:36:16.609 metplus INFO: COMMAND: /glade/p/ral/jntp/MET/MET_releases/10.0.0/bin/series_analysis -fcst /glade/u/home/nleonardo/METplus_Tutorial/ATTEMPT001/output/series_analysis_lead/series_F000/FCST_FILES_F000 -obs /glade/u/home/nleonardo/METplus_Tutorial/ATTEMPT001/output/series_analysis_lead/series_F000/OBS_FILES_F000 -out /glade/u/home/nleonardo/METplus_Tutorial/ATTEMPT001/output/series_analysis_lead/series_F000/series_F000_PBLH_(,).nc -paired -config /glade/p/ral/jntp/MET/METplus/METplus-4.0.0/parm/met_config/SeriesAnalysisConfig_wrapped -v 210/14 13:36:16.610 metplus (command_runner.py:124) DEBUG: log_name is: series_analysis, output sent to: /glade/u/home/nleonardo/METplus_Tutorial/ATTEMPT001/output/logs/metplus.log.20211014133613
OUTPUT:
DEBUG 1: Default Config File: /glade/p/ral/jntp/MET/MET_releases/10.0.0/share/met/config/SeriesAnalysisConfig_defaultDEBUG 1: User Config File: /glade/p/ral/jntp/MET/METplus/METplus-4.0.0/parm/met_config/SeriesAnalysisConfig_wrapped
GSL_RNG_TYPE=mt19937
GSL_RNG_SEED=18446744072376182881
DEBUG 1: Length of configuration "fcst.field" = 1
DEBUG 1: Length of configuration "obs.field" = 1
DEBUG 1: Length of forecast file list = 1
DEBUG 1: Length of observation file list = 1
DEBUG 1: The "fcst.field" and "obs.field" configuration entries and the "-fcst" and "-obs" command line options all have length one.
DEBUG 2: Processing series entry 1 of 1: PBLH(,) versus PBLH(,)
WARNING:
WARNING: get_series_entry() -> No match found for PBLH(,) in file: /glade/u/home/nleonardo/METplus_Tutorial/ATTEMPT001/output/extract_tiles/20210709_12/AL052021/FCST_TILE_F000_Wgfs4_PBLH_20210709_1200_000.nc
WARNING:
ERROR :
ERROR : DataPlane::set_constant(double) -> no data buffer allocated!
ERROR :
...

This is all just a simple test in which I am strictly doing a one-to-one comparison between two sets of files from 1 initialization cycle (e.g., comparing the same forecast hours/ valid times between sets). Hence, I'm not concerned about complex comparisons in which I validate different hours between sets.

Thank you.

[georgemccabe]

This use case uses tc_stat output to determine the storm track, then creates cutouts (40x40) of the data around the storm center. The empty files may be a result of a mismatch between the storm location and the grid of the data. I am working limited days in October, but I will log into Cheyenne when I have a chance and take a closer look to see where the disconnect is occurring.

[georgemccabe]

OK, I looked at the use case on Cheyenne and the issue is the value that is set for the offset to apply to the lat/lon points of the storm track to calculate the lower left corner. You had the following set:

EXTRACT_TILES_LON_ADJ = 10
EXTRACT_TILES_LAT_ADJ = 10

The domain of the input WRF files are about 3.5 by 8 degrees, so subtracting 10 from the lat/lon points of the storm track (i.e. 40.3/-73.1) resulted in a lower left corner that was way off of the input grid. I changed these values to 1. I also slightly adjusted the dx/dy values set by EXTRACT_TILES_DLAT/LON to 0.05 to match the nx/ny (40/40) and new offset values:

EXTRACT_TILES_NLAT = 40
EXTRACT_TILES_NLON = 40

EXTRACT_TILES_DLAT = 0.05
EXTRACT_TILES_DLON = 0.05

EXTRACT_TILES_LON_ADJ = 1
EXTRACT_TILES_LAT_ADJ = 1

We have discussed removing the EXTRACT_TILES_[LON/LAT]_ADJ in favor of simply calculating the offset based on the values of NLAT/NLON/DLAT/LON to prevent this type of mistake.

With these adjustments I was able to get extract tiles output that had valid data and SeriesAnalysis ran without error. I noticed the your SERIES_ANALYSIS_OUTPUT_TEMPLATE value referenced {fcst_level}, which is a NetCDF level value, i.e. (*,*), that doesn't work well in output filenames. I removed references to that variable to prevent that. The animation files that are generated appear to be hard-coded to include the level value, so improvements to that logic to prevent the use of the wildcard * character could be useful.

You can see my modified conf file on Cheyenne here:

/glade/u/home/mccabe/TCStat_SeriesAnalysis_wrfPBLH_test.conf

and the output from my run here:

/glade/u/home/mccabe/data/out-elsa

[georgemccabe]

Hi @nleonardo87, have you had a chance to look my suggestions above? I'd like to mark this discussion as being answered and "lock it" to prevent future posts. That's how we encourage users to create new discussions for new questions. But I wanted to give you an opportunity to comment on it before doing so. Please feel free to select one of the responses as being the best answer to your original question. That'll help future users with similar questions find answers.

[nleonardo87]

Thank you. It looks to have worked.

[georgemccabe]

Great, glad to hear it! I am going to lock this discussion to prevent future comments, but please feel free to create a new discussion topic if you have any other questions.