Output within Pointstat mpr files

[robdarvell]

Question around the use of PointStat and the storage of the raw matched pair data within mpr files (or perhaps it should be elsewhere)

We store the match pair data within one of our databases. For the Surface fields (as an example given we do it for all ob types) we require to be able to distinguish between those which are manual obs and those which are automatic obs.
Within my ascii file which I then use ascii2nc on I use the Message type column (col1) to refer to the different types. So for example you might see AUTO and MANU rather than simply SURFACE. Couldn’t work out whether there was a better column to put this information into.

Within my PointStat job I have created a message_type_group_map with a key of say SURF and then with the values of AUTO, MANU. I then refer to that in my message_type keyword.

Job runs fine, but in the output mpr file, in the OBTYPE column it just refers to SURF all the time, and doesn’t show the distinction between the elements, which is what I want. In the point_stat.cc code I can see where this is set:
// Store the message type in the obtype column
shc.set_obtype(conf_info.vx_opt[i].msg_typ[j].c_str());

So, what can I do next?
I thought, I could set up multiple key/value message type group maps, one for AUTO and one for MANU. Did this, re-ran the code and low and behold, in the OBTYPE column I now have AUTO and MANU.
You may be thinking now, so why have you asked this in a discussion?

There is one more problem that I face, which I am trying to avoid having to do all the duplicate checking outside of the MET code, as that is doing the satisfactory duplicate checks for this example within the code.

Due to how I run the job I need to use a time window in which to extract the obs (for example 2hrs before and after the time I wish to verify to try and maximise the amount of data I use). Some locations report a few times in that period and the MET code does work out and only include the one closest to the valid time I have asked for. All good.

However, some report a manual ob at some times and then an automatic at others in the period. When I am using the 2 different keys approach I am therefore getting some obs in twice, one in the manual and one in the auto section, whereas when I cover both options in the message_type I just get the one ob closest to the valid time (which is what we require).

Is there anyway of getting the message_type value to be included in the OBTYPE column rather than the message_type key in the output ? Could there be an over-ride to what is in the code at present (so that it doesn’t affect others?), or, is there another method I could have used which I haven’t considered.

[JohnHalleyGotway]

@robdarvell we ran across this very old discussion that we never addressed. Although we may have discussed this during one of the MetOffice developer telecons.

Can you please take a quick look and let me know if questions remain from this discussion that should be addressed? Or if not, I could just delete it.

Please let me know how you'd like to proceed.

[robdarvell]

@JohnHalleyGotway : Yes, this is still something which would be preferable if it could be looked into when convenient. I have a version of the MET code where I have added a new element within pair_base.cc and then updating the mpr file with that variable, which is doing what I was hoping it would do. However, I have not seen whether there are any unforeseen circumstance from other things which I may then have broken.

[JohnHalleyGotway]

@robdarvell please take a look at this newly created dtcenter/MET#2893 issue and confirm that it describes the functionality that's actually needed.

[robdarvell]

@JohnHalleyGotway : Yes, that covers what I have requested, and in a way which allows the majority to continue 'as is' without needing to change anything.
I have also just added a comment to the issue, just giving a small bit of extra context, in case the question, 'why couldn't you ....' was asked.

[JohnHalleyGotway]

Thanks Rob. I marked this discussion as being answered and recommend that we add any future comments directly on issue dtcenter/MET#2893. Thanks!