Handle some special cases for wrf-python #218

zmoon · 2024-11-16T00:11:43Z

The "uvmet" variables and a few other tweaks

@quaz115 let me know if it doesn't work for you

for wind speed/direction variables rotated to earth coords

Copilot reviewed 1 out of 1 changed files in this pull request and generated no suggestions.

quaz115 · 2024-11-16T03:20:47Z

The "uvmet" variables

@quaz115 let me know if it doesn't work for you

@zmoon I am getting this issue as i tried it:

self.obj = mio.models._wrfchem_mm.open_mfdataset(self.files,**self.mod_kwargs)

File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/monetio/monetio/models/_wrfchem_mm.py", line 139, in open_mfdataset
dset = xr.merge(var_wrf_list)
File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 976, in merge
merge_result = merge_core(
File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 701, in merge_core
variables, out_indexes = merge_collected(
File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 290, in merge_collected
merged_vars[name] = unique_variable(
File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 144, in unique_variable
raise MergeError(
xarray.core.merge.MergeError: conflicting values for variable 'wspd_wdir' on objects to be combined. You can skip this check by specifying compat='override'.

quaz115 · 2024-11-16T03:26:50Z

The "uvmet" variables
@quaz115 let me know if it doesn't work for you

@zmoon I am getting this issue as i tried it:
self.obj = mio.models._wrfchem_mm.open_mfdataset(self.files,**self.mod_kwargs)
File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/monetio/monetio/models/_wrfchem_mm.py", line 139, in open_mfdataset dset = xr.merge(var_wrf_list) File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 976, in merge merge_result = merge_core( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 701, in merge_core variables, out_indexes = merge_collected( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 290, in merge_collected merged_vars[name] = unique_variable( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 144, in unique_variable raise MergeError( xarray.core.merge.MergeError: conflicting values for variable 'wspd_wdir' on objects to be combined. You can skip this check by specifying compat='override'.

For YAML obs key added these:

obs:
  pasadena_ground: # obs label
    filename: '/scratch2/BMC/rcm1/qrasool/wrfout_postprocessing/LA_Ground_merge_NEW21Sept2023Update.csv' #'/wrk/users/charkins/MM_data/pasadena_ground_ict/20210725/recap-CIT-AMS1-NR-PM1_CITLL_20210725_RB.ict'
    obs_type: ground
    time_var: 'MasterTimewave_Start_UTC'
    ground_coordinate: {'latitude':34.136363,'longitude':-118.126817}
    #resample: '600S' #10 min so works on Hera as a test. Can comment this if submitting a job. 
    variables: #Opt 
      'uvmet10_wspd':
        unit_scale: 1 #Opt Scaling factor 
        unit_scale_method: '*' #'/' #'*' #Opt Multiply = '*' , Add = '+', subtract = '-', divide = '/'
        ylabel_plot: 'Wind speed (m s^-1)'
      'uvmet10_wdir':
        unit_scale: 1 #Opt Scaling factor 
        unit_scale_method: '*' #'/' #'*' #Opt Multiply = '*' , Add = '+', subtract = '-', divide = '/'
        ylabel_plot: 'Wind dir (degrees)'

quaz115 · 2024-11-16T03:43:01Z

The "uvmet" variables
@quaz115 let me know if it doesn't work for you

@zmoon I am getting this issue as i tried it:
self.obj = mio.models._wrfchem_mm.open_mfdataset(self.files,**self.mod_kwargs)
File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/monetio/monetio/models/_wrfchem_mm.py", line 139, in open_mfdataset dset = xr.merge(var_wrf_list) File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 976, in merge merge_result = merge_core( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 701, in merge_core variables, out_indexes = merge_collected( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 290, in merge_collected merged_vars[name] = unique_variable( File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 144, in unique_variable raise MergeError( xarray.core.merge.MergeError: conflicting values for variable 'wspd_wdir' on objects to be combined. You can skip this check by specifying compat='override'.
For YAML obs key added these: obs: pasadena_ground: # obs label filename: '/scratch2/BMC/rcm1/qrasool/wrfout_postprocessing/LA_Ground_merge_NEW21Sept2023Update.csv' #'/wrk/users/charkins/MM_data/pasadena_ground_ict/20210725/recap-CIT-AMS1-NR-PM1_CITLL_20210725_RB.ict' obs_type: ground time_var: 'MasterTimewave_Start_UTC' ground_coordinate: {'latitude':34.136363,'longitude':-118.126817} #resample: '600S' #10 min so works on Hera as a test. Can comment this if submitting a job. variables: #Opt 'uvmet10_wspd': unit_scale: 1 #Opt Scaling factor unit_scale_method: '' #'/' #'' #Opt Multiply = '' , Add = '+', subtract = '-', divide = '/' ylabel_plot: 'Wind speed (m s^-1)' 'uvmet10_wdir': unit_scale: 1 #Opt Scaling factor unit_scale_method: '' #'/' #'' #Opt Multiply = '' , Add = '+', subtract = '-', divide = '/' ylabel_plot: 'Wind dir (degrees)'

@zmoon YAML path on Hera: /scratch2/BMC/rcm1/qrasool/wrfout_postprocessing/control_wrfchem_groundsunvex_QZR_mmdevelopaircraft_Copy_Rocky8_SSMATOMo3_LowvsHighNox_RefinedPlotting.yaml

quaz115 · 2024-11-16T20:26:40Z

@zmoon still the same issue: m.open_model_files(time_interval=time_interval, control_dict=self.control_dict)
File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/MELODIES-MONET/melodies_monet/driver.py", line 566, in open_model_files
self.obj = mio.models._wrfchem_mm.open_mfdataset(self.files,**self.mod_kwargs)
File "/scratch2/BMC/rcm1/qrasool/MMdir_developSept2024/monetio/monetio/models/_wrfchem_mm.py", line 139, in open_mfdataset
dset = xr.merge(var_wrf_list)
File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 976, in merge
merge_result = merge_core(
File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 701, in merge_core
variables, out_indexes = merge_collected(
File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 290, in merge_collected
merged_vars[name] = unique_variable(
File "/scratch2/BMC/rcm1/qrasool/anaconda3/envs/mm_developaircraftSept2024/lib/python3.9/site-packages/xarray/core/merge.py", line 144, in unique_variable
raise MergeError(
xarray.core.merge.MergeError: conflicting values for variable 'wspd_wdir' on objects to be combined. You can skip this check by specifying compat='override'.

quaz115 · 2024-11-18T06:05:59Z

@zmoon I don't have permission to push my commit, but i made following updates to the wrf-chem reader py file and it is working now (for rh (%), wspd: uvmet_wspd or uvmet10_wspd (m/s), wdir: uvmet_wdir or uvmet10_wdir (degrees)), can you review this (@rschwant for reference):

    wrflist = []
    for files in fname:
        wrflist.append(Dataset(files))

    if not surf_only_nc:
        # Add some additional defaults needed for aircraft analysis
        # Turn this on also if need to convert aerosols
        var_list.append("pres")
        var_list.append("pressure") 
        var_list.append("height")
        var_list.append("tk")
        var_list.append("temp")  
        var_list.append("rh")  
        var_list.append("height_agl")
        var_list.append("PSFC")
        # need to calculate surface pressure and dp and optionally dz here.

        # Additional defaults for satellite analysis
        var_list.append("zstag")

    # Remove 'uvmet10_wspd' and 'uvmet10_wdir' if present
    # Adjust var_list for wind variables
    if any(v in {"uvmet10_wspd", "uvmet10_wdir"} for v in var_list):
        # Remove 'uvmet10_wspd' and 'uvmet10_wdir' from var_list
        var_list = [v for v in var_list if v not in {"uvmet10_wspd", "uvmet10_wdir"}]
        # Add 'uvmet10_wspd_wdir' to var_list
        var_list.append("uvmet10_wspd_wdir")

    var_wrf_list = []
    for var in var_list:
        if var == "pres":
            # Handle 'pres' variable
            var_wrf = getvar(
                wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False, units="Pa"
            )
            var_wrf = var_wrf.rename(var)
            var_wrf_list.append(var_wrf)
            continue
        elif var in {"height", "height_agl", "zstag", "temp", "rh"}:
            if var in {"height", "height_agl", "zstag"}:
                # For height variables, specify units
                var_wrf = getvar(
                    wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False, units="m"
                )
            else:
                # For 'temp' and 'rh', use default units
                var_wrf = getvar(
                    wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False
                )
            # Remove 'wspd_wdir' from coordinates if present
            if 'wspd_wdir' in var_wrf.coords:
                print(f"Removing 'wspd_wdir' from coordinates of variable '{var}'")
                var_wrf = var_wrf.reset_coords('wspd_wdir', drop=True)
            var_wrf = var_wrf.rename(var)
            var_wrf_list.append(var_wrf)
            continue
        elif var in {"uvmet10", "uvmet"}:
            # Existing code for wind variables
            pref = var.split("_")[0]
            var_wrf = getvar(
                wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False
            )
            var_wrf_list.extend(
                [
                    var_wrf.isel(u_v=0).rename(f"{pref}_u"),
                    var_wrf.isel(u_v=1).rename(f"{pref}_v"),
                ]
            )
            continue
        elif var in {"uvmet10_wspd_wdir", "uvmet_wspd_wdir"}:
            # Existing code for wind speed and direction
            pref = var.split("_")[0]
            var_wrf = getvar(
                wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False
            )
            var_wrf_list.extend(
                [
                    var_wrf.isel(wspd_wdir=0).rename(f"{pref}_wspd"),
                    var_wrf.isel(wspd_wdir=1).rename(f"{pref}_wdir").assign_attrs(units="deg"),
                ]
            )
            continue
        else:
            # Existing code for other variables
            var_wrf = getvar(
                wrflist, var, timeidx=ALL_TIMES, method="cat", squeeze=False
            )
            # Remove 'wspd_wdir' from coordinates if present
            if 'wspd_wdir' in var_wrf.coords:
                print(f"Removing 'wspd_wdir' from coordinates of variable '{var}'")
                var_wrf = var_wrf.reset_coords('wspd_wdir', drop=True)
            var_wrf = var_wrf.rename(var)
            var_wrf_list.append(var_wrf)
        print(f"Fetched variable '{var}' with internal name '{var_wrf.name}'")
    
    # Remove 'wspd_wdir' from coordinates of all variables before merging
    for idx, var_data in enumerate(var_wrf_list):
        if 'wspd_wdir' in var_data.coords:
            print(f"Removing 'wspd_wdir' from coordinates of variable '{var_data.name}'")
            var_data = var_data.reset_coords('wspd_wdir', drop=True)
            var_wrf_list[idx] = var_data
    
    # Proceed with the merge
    dset = xr.merge(var_wrf_list)

quaz115 · 2024-11-18T16:40:38Z

@zmoon can you also add 'rh' as well (i added it for my case). I think, we should have these MET variables: T, P, RH, WSPD, WDIR for sure from getvar utility

zmoon · 2024-11-18T16:58:08Z

@quaz115 I'm pretty sure 'tk' (already in the aircraft list) is the same is 'temp' with default units. Would you say aircraft always needs RH? If not, would it work to just add it to the var list in your YAML section?

quaz115 · 2024-11-18T17:51:42Z

@quaz115 I'm pretty sure 'tk' (already in the aircraft list) is the same is 'temp' with default units. Would you say aircraft always needs RH? If not, would it work to just add it to the var list in your YAML section?

@zmoon Sorry for the confusion, i meant in general (about these variables that should be there for MET obs vs model comparisons, for future as well, and PBLH additionally). Also Thanks for the simplified code fixes. Just re-ran with this branch and it works well. Please go ahead with the Merge

MM standard variable for model temperature

rschwant · 2024-11-18T21:02:03Z

Temperature update fixes the problem I was seeing!

zmoon · 2024-11-20T23:11:10Z

@quaz115 @rschwant any final thoughts?

rschwant · 2024-11-20T23:20:29Z

Looks good to me!

quaz115 · 2024-11-21T02:35:51Z

@quaz115 @rschwant any final thoughts?

Looks good to be merged for me as well

zmoon added 3 commits November 15, 2024 09:24

Implement some special cases

d32ba96

for wind speed/direction variables rotated to earth coords

pref not needed for this one

aabe31a

Set units for wind dir

a60fdfc

zmoon requested a review from Copilot November 16, 2024 00:11

Copilot AI reviewed Nov 16, 2024

View reviewed changes

zmoon added 2 commits November 18, 2024 10:22

Simplify pref code for uv

29293f7

Remove unwanted coordinate variables

01e5473

zmoon added 3 commits November 18, 2024 13:58

Some .name comments

3911467

Mutable default arg

898528b

Allow requesting 'temperature_k'

50eaa37

MM standard variable for model temperature

zmoon force-pushed the wrf-wspd-wdir branch from cc33d00 to 50eaa37 Compare November 18, 2024 20:20

Just skip

558cf3d

zmoon force-pushed the wrf-wspd-wdir branch from 3334132 to 558cf3d Compare November 18, 2024 20:35

zmoon marked this pull request as ready for review November 18, 2024 21:13

zmoon merged commit 798b086 into noaa-oar-arl:develop Nov 21, 2024
7 checks passed

zmoon deleted the wrf-wspd-wdir branch November 21, 2024 03:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle some special cases for wrf-python #218

Handle some special cases for wrf-python #218

zmoon commented Nov 16, 2024 •

edited

Loading

quaz115 commented Nov 16, 2024

quaz115 commented Nov 16, 2024 •

edited by zmoon

Loading

quaz115 commented Nov 16, 2024

quaz115 commented Nov 16, 2024 •

edited

Loading

quaz115 commented Nov 18, 2024

quaz115 commented Nov 18, 2024

zmoon commented Nov 18, 2024

quaz115 commented Nov 18, 2024

rschwant commented Nov 18, 2024

zmoon commented Nov 20, 2024

rschwant commented Nov 20, 2024

quaz115 commented Nov 21, 2024

Handle some special cases for wrf-python #218

Handle some special cases for wrf-python #218

Conversation

zmoon commented Nov 16, 2024 • edited Loading

Choose a reason for hiding this comment

quaz115 commented Nov 16, 2024

quaz115 commented Nov 16, 2024 • edited by zmoon Loading

quaz115 commented Nov 16, 2024

quaz115 commented Nov 16, 2024 • edited Loading

quaz115 commented Nov 18, 2024

quaz115 commented Nov 18, 2024

zmoon commented Nov 18, 2024

quaz115 commented Nov 18, 2024

rschwant commented Nov 18, 2024

zmoon commented Nov 20, 2024

rschwant commented Nov 20, 2024

quaz115 commented Nov 21, 2024

zmoon commented Nov 16, 2024 •

edited

Loading

quaz115 commented Nov 16, 2024 •

edited by zmoon

Loading

quaz115 commented Nov 16, 2024 •

edited

Loading