Adjustments to allow ingest of non-traditional data into METplus.

[jwraby]

I enjoyed the METplus Advanced Training Series presentations yesterday. In Amanda's presentation on Fire Weather, I noted a comment with wording similar to the title I used for this discussion. Reformatting data into a MET-compatible format can sometimes be a challenge. I have some sensor data in text files which are formatted so that each row contains observations of met variables for a specific time. The sampling rate ranges from several per second for wind component data and 1 per second for the other met variables. I will need to average the data in 15-min bins to match with the forecast model output. My first thought was to reformat the averaged data into the MET 11-col format so it could be ingested into MET Ascii2nc and then Point-Stat. I envisioned having to do the averaging and reformatting this using Excel spreadsheets, but wondered if there is a way to do this using METplus? I can't get into more specifics about the data, but am interested in learning about any ideas you might have on how to approach the processing of this data for ingest into MET and METplus.

[j-opatz]

Hi John,

And thank you for your question. I'm glad you asked, because there is a very simple way to do temporal averaging on point data in ASCII2NC.

As you first noted, you will need to construct a Python script to read in your text-based dataset and pass the METplus-readable 11 column dataset back to METplus. Here's a link to a description of that 11 column dataset (make sure you pay special attention to the dtype for each of those variables!), as well as a recent METplus use case that uses Python to pull data from a netCDF that is non-native to METplus and place it into the 11 column format. While your situation will require tailoring the Python script ingest method to your rows of data (the example reads in an entire directory of netCDFs), it might help to see some of the commands that were used to create a list of lists, which is the necessary format. Alternatively, just below the 11 column description in the MET User's Guide is a very simple method for convert a Pandas dataframe into the list format; if you're already using Pandas this is definitely the way to go!

As for the temporal interpolation, the MET User's guide provides some tips on how to use the time_summary dictionary in ASCII2NC, which lets a user fine-tune the time summary options for point data. In your example, getting 15 minute bins would have width set to 900 and the type list to include "mean" to compute the mean time summary. From there, you would need to determine how often you want to summarize the data (using step) and the time limits on your data (using beg and end).

Please let me know if you have any more questions on this topic!

[jwraby]

John - Thank you for your reply with some useful suggestions on how I could approach the reformatting and averaging tasks for my data. It would have taken a lot of time for me to come up with these ideas on my own (assuming that I would have ever come up with the same ideas instead of heading off in some other direction less likely to succeed) and for that I really appreciate your help which has provided some insight into the power of METplus for performing similar tasks. I will review the link you provided and see if I can apply you suggestions to my data.