hard to understand behavior of concat

[ghost]

I have two datasets (one main dataset, with a second dataset which I want to append to it) and want to concatenate them along one axis. It seems like a straightforward call to xr.concat(), but I'm getting a result that I find hard to understand.

Here is the main dataset "dsnam"

<xarray.Dataset>
Dimensions:   (Xind: 10, Yind: 17, tZ: 23744)
Coordinates:
  * tZ        (tZ) datetime64[ns] 2004-03-01 ... 2020-05-31T18:00:00
  * Yind      (Yind) int64 240 241 242 243 244 245 ... 251 252 253 254 255 256
  * Xind      (Xind) int64 516 517 518 519 520 521 522 523 524 525
Data variables:
    u10m      (tZ, Yind, Xind) float64 nan nan nan nan nan ... nan nan nan nan
    v10m      (tZ, Yind, Xind) float64 nan nan nan nan nan ... nan nan nan nan
    gap       (tZ) float64 1.0 1.0 1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0 1.0 1.0
    Lat       (Yind, Xind) float64 41.83 41.82 41.8 41.78 ... 43.34 43.32 43.3
    Lon       (Yind, Xind) float64 -71.23 -71.09 -70.95 ... -69.85 -69.71 -69.56
    narrfill  (tZ) float64 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0

(Although it looks like the main variables u10m and v10m are all NaNs, they aren't-- only the beginning/ending groups of them are.)

Here is the smaller dataset I want to append "dsnamnew":

<xarray.Dataset>
Dimensions:   (Xind: 10, Yind: 17, tZ: 856)
Coordinates:
  * tZ        (tZ) datetime64[ns] 2020-06-01 ... 2020-12-31T18:00:00
  * Yind      (Yind) int64 240 241 242 243 244 245 ... 251 252 253 254 255 256
  * Xind      (Xind) int64 516 517 518 519 520 521 522 523 524 525
Data variables:
    u10m      (tZ, Yind, Xind) float64 1.601 1.572 1.524 ... 9.669 9.883 9.841
    v10m      (tZ, Yind, Xind) float64 -4.582 -4.736 -5.084 ... -2.312 -2.33
    gap       (tZ) float64 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0
    narrfill  (tZ) float64 1.0 1.0 1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0 1.0 1.0
    Lat       (Yind, Xind) float64 41.83 41.82 41.8 41.78 ... 43.34 43.32 43.3
    Lon       (Yind, Xind) float64 -71.23 -71.09 -70.95 ... -69.85 -69.71 -69.56

I want to concatenate along the dimension "tZ" so I make this call:

both = xr.concat( [ dsnam, dsnamnew ], dim='tZ')

And here is the result "both":

<xarray.Dataset>
Dimensions:   (Xind: 10, Yind: 17, tZ: 24600)
Coordinates:
  * tZ        (tZ) datetime64[ns] 2004-03-01 ... 2020-12-31T18:00:00
  * Yind      (Yind) int64 240 241 242 243 244 245 ... 251 252 253 254 255 256
  * Xind      (Xind) int64 516 517 518 519 520 521 522 523 524 525
Data variables:
    u10m      (tZ, Yind, Xind) float64 nan nan nan nan ... 9.669 9.883 9.841
    v10m      (tZ, Yind, Xind) float64 nan nan nan nan ... -2.348 -2.312 -2.33
    gap       (tZ) float64 1.0 1.0 1.0 1.0 1.0 1.0 ... 0.0 0.0 0.0 0.0 0.0 0.0
    Lat       (tZ, Yind, Xind) float64 41.83 41.82 41.8 ... 43.34 43.32 43.3
    Lon       (tZ, Yind, Xind) float64 -71.23 -71.09 -70.95 ... -69.71 -69.56
    narrfill  (tZ) float64 0.0 0.0 0.0 0.0 0.0 0.0 ... 1.0 1.0 1.0 1.0 1.0 1.0

All aspects are just the way I expected ... except that Lat and Lon are now (tZ,Yind,Xind) instead of (Yind,Xind).

Why would this happen?

[andersy005]

@DanCodigaMWRA

Why would this happen?

You want to treat Lat and Lon as coordinates during the concatenation, however with your current setup, xarray isn't aware of this fact. There are two remedies to this.

• Specify data_vars i.e. xr.concat(...., data_vars=['v10m, 'u10m', ...])

In [13]: ds
Out[13]: 
<xarray.Dataset>
Dimensions:  (Xind: 2, Yind: 3, tZ: 3)
Coordinates:
  * Xind     (Xind) int64 0 1
  * Yind     (Yind) int64 0 1 2
  * tZ       (tZ) datetime64[ns] 2021-01-01 2021-02-01 2021-03-01
Data variables:
    Lat      (Xind, Yind) int64 0 1 2 3 4 5
    u10m     (tZ, Xind, Yind) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17


In [15]: xr.concat([ds, ds], dim='tZ', data_vars=['u10m'])
Out[15]: 
<xarray.Dataset>
Dimensions:  (Xind: 2, Yind: 3, tZ: 6)
Coordinates:
  * Xind     (Xind) int64 0 1
  * Yind     (Yind) int64 0 1 2
  * tZ       (tZ) datetime64[ns] 2021-01-01 2021-02-01 ... 2021-02-01 2021-03-01
Data variables:
    Lat      (Xind, Yind) int64 0 1 2 3 4 5
    u10m     (tZ, Xind, Yind) int64 0 1 2 3 4 5 6 7 ... 10 11 12 13 14 15 16 17

• Set Lat, Lon to coordinates via set_coords(['Lat', 'Lon']) prior to the concatenation

In [16]: xr.concat([ds.set_coords(['Lat']), ds.set_coords(['Lat'])], dim='tZ')
Out[16]: 
<xarray.Dataset>
Dimensions:  (Xind: 2, Yind: 3, tZ: 6)
Coordinates:
  * Xind     (Xind) int64 0 1
  * Yind     (Yind) int64 0 1 2
  * tZ       (tZ) datetime64[ns] 2021-01-01 2021-02-01 ... 2021-02-01 2021-03-01
    Lat      (Xind, Yind) int64 0 1 2 3 4 5
Data variables:
    u10m     (tZ, Xind, Yind) int64 0 1 2 3 4 5 6 7 ... 10 11 12 13 14 15 16 17

I hope this helps

[ghost]

Nice. I think you just helped me understand that I should be treating Lat and Lon as coordinates (not data variables) throughout my analysis! Thank you!