add get_partition_input #289

juliettelavoie · 2023-11-17T17:57:26Z

Pull Request Checklist:

This PR addresses an already opened issue (for bug fixes / features)
- This PR fixes #xyz
(If applicable) Documentation has been added / updated (for bug fixes / features).
(If applicable) Tests have been added.
This PR does not seem to break the templates.
HISTORY.rst has been updated (with summary of main changes).
- Link to issue (:issue:number) and pull request (:pull:number) has been added.

What kind of change does this PR introduce?

Add xs.ensembles.get_partition_input to prepare data from catalog to be passed to xclim partionning functions (https://xclim.readthedocs.io/en/stable/api.html#uncertainty-partitioning)
Now, I am wondering if it would be more xscen to also wrap the xclim fonction instead only prepping the data? This would allow to add a attrs processing_level= partition to be able to put it in a a catalog.
workflow 1: xs.get_partition_input -> xc.ensembles.lafferty_sriver -> fg.partition
workflow 2: xs.partition (which gets the input and wraps xclim) -> maybe save to disk and catalog -> fg.partition or other plots (maybe maps of one component)
EDIT: I went with workflow 1

Does this PR introduce a breaking change?

no

Other information:

I wasn't sure if this should be in extract.py (because we extract from a cat) or in ensembles.py ( to match the xclim module).
companion de Add Lafferty and Sriver partition xclim#1529 et add partition plot figanos#134

review-notebook-app · 2023-12-12T16:33:38Z

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

# Conflicts: # CHANGES.rst

### Pull Request Checklist: - [x] This PR addresses an already opened issue (for bug fixes / features) - This PR fixes #1497 - [x] Tests for the changes have been added (for bug fixes / features) - [x] (If applicable) Documentation has been added / updated (for bug fixes / features) - [x] CHANGES.rst has been updated (with summary of main changes) - [x] Link to issue (:issue:`number`) and pull request (:pull:`number`) has been added ### What kind of change does this PR introduce? * Add partition algo from Lafferty and Sriver (2023) ### Does this PR introduce a breaking change? depends (see options below) ### Other information: TODO list in this PR: - [x] Add weights - [x] add tests - [x] Maybe add `num` to `hawkins_sutton` ? - [x] Clean up based on changes outside of this PR (eg. remove functions that are now elsewhere) TODO list outside of this PR: - [ ] Add a more general `graph_fraction_of_total_variance` to figanos (in progress by Juliette, Ouranosinc/figanos#134) - [ ] Add a function in xscen to prepare the data from a catalog. (Ouranosinc/xscen#289) In this function, I could rename the dimension to fit what the partition vocabulary (`model`, `scenario`, `downscaling`) OR we could change the partition vocab to the catalog/xscen vocab (`source`, `experiment`, `bias_adjust_project`). Option 2 is my personal preference, but that would be a breaking change for `hawkins_sutton`.

…ract

…ract # Conflicts: # docs/notebooks/4_ensembles.ipynb

for more information, see https://pre-commit.ci

juliettelavoie · 2023-12-21T13:09:28Z

I don't understand what's going on. In commit, 2ece1d7 everything passed. I removed inconsequential lines andt the RTD build keeps failing. (I can run the notebook no problem on our servers). I have tried to rebuild the rtd a few times. In these different builds (without changing anything), the error is not always the same. Either the datasets doesn't load with a HDF5 error or it loads an empty dataset and fail when the function tries to subset ('Dataset' object has no attribute 'lon').

HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
  #000: H5A.c line 1891 in H5Aiterate2(): invalid location identifier
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: H5VLint.c line 1741 in H5VL_vol_object(): invalid identifier type to function
    major: Invalid arguments to routine
    minor: Inappropriate type
ERROR:traitlets:Kernel died while waiting for execute reply.

HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
  #000: H5A.c line 1891 in H5Aiterate2(): invalid location identifier
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: H5VLint.c line 1741 in H5VL_vol_object(): invalid identifier type to function
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
  #000: H5A.c line 1891 in H5Aiterate2(): invalid location identifier
    major: Invalid arguments to routine
    minor: Inappropriate type
  #001: H5VLint.c line 1741 in H5VL_vol_object(): invalid identifier type to function
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
  #000: H5D.c line 767 in H5Dget_create_plist(): invalid dataset identifier
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.14.3) thread 0:
  #000: H5D.c line 481 in H5Dclose(): can't decrement count on dataset ID
    major: Dataset
    minor: Unable to decrement reference count
  #001: H5Iint.c line 1227 in H5I_dec_app_ref_always_close(): can't decrement ID ref count
    major: Object ID
    minor: Unable to decrement reference count
  #002: H5Iint.c line 1197 in H5I__dec_app_ref_always_close(): can't decrement ID ref count
    major: Object ID
    minor: Unable to decrement reference count
  #003: H5Iint.c line 946 in H5I_remove(): can't remove ID node
    major: Object ID
    minor: Can't delete message
  #004: H5Iint.c line 895 in H5I__remove_common(): can't remove ID node from hash table
    major: Object ID
    minor: Can't delete message
  #005: H5Iint.c line 1077 in H5I__dec_app_ref(): can't decrement ID ref count
    major: Object ID
    minor: Unable to decrement reference count
  #006: H5Iint.c line 981 in H5I__dec_ref(): can't locate ID
    major: Object ID
    minor: Unable to find ID information (already closed?)

huard

Pas assez familier avec xscen pour faire une révision en profondeur, mais ça me semble bon.

huard · 2024-01-08T19:22:20Z

docs/notebooks/4_ensembles.ipynb

+   "metadata": {},
+   "source": [
+    "## Ensemble partition\n",
+    "This tutorial will show how to use the xscen to create the input for [xclim partition functions](https://xclim.readthedocs.io/en/stable/api.html#uncertainty-partitioning)."


Suggested change

"This tutorial will show how to use the xscen to create the input for [xclim partition functions](https://xclim.readthedocs.io/en/stable/api.html#uncertainty-partitioning)."

"This tutorial will show how to use xscen to create the input for [xclim partition functions](https://xclim.readthedocs.io/en/stable/api.html#uncertainty-partitioning)."

RondeauG

LGTM! One thing that is not currently supported (but it's maybe not an issue) is regional models. If you want to delve into that, you could look at:

xscen/xscen/ensembles.py

Line 374 in ce54df8

for k in info:

The idea is to look at driving_model instead of source, and fill that entry with source in the case of global models.

RondeauG · 2024-01-09T14:31:05Z

docs/notebooks/4_ensembles.ipynb

+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The function searches the catalog with `search_kw` and creates a dataset with new dimensions in `partition_dim`(`[\"source\", \"experiment\", \"bias_adjust_project\"]`). \n",


This will need to be edited, since it still relates to the previous behaviour.

juliettelavoie · 2024-01-09T15:56:46Z

LGTM! One thing that is not currently supported (but it's maybe not an issue) is regional models. If you want to delve into that, you could look at:

xscen/xscen/ensembles.py

Line 374 in ce54df8

for k in info:

The idea is to look at driving_model instead of source, and fill that entry with source in the case of global models.

I don't think this is necessary for now. I will do another PR if that changes.

…partition-extract

add get_partition_input

7f1da4c

juliettelavoie marked this pull request as draft November 17, 2023 17:57

juliettelavoie mentioned this pull request Nov 17, 2023

Add Lafferty and Sriver partition Ouranosinc/xclim#1529

Merged

11 tasks

juliettelavoie added 2 commits December 11, 2023 16:38

fix cftime + rename_dict

1e73589

add doc example

c4c17b9

github-actions bot added the docs label Dec 12, 2023

juliettelavoie added 6 commits December 12, 2023 11:37

remove comment

9bda2c3

improve doc

a8898d9

add test

146dc10

Merge remote-tracking branch 'origin/main' into add-partition-extract

5b57c8d

# Conflicts: # CHANGES.rst

comments

9171778

Merge remote-tracking branch 'origin/main' into add-partition-extract

61287be

# Conflicts: # CHANGES.rst

juliettelavoie added 8 commits December 14, 2023 14:36

fix test

deb00f6

fix doc

58dc200

test to get make doc pass

3aaeb01

try to remove wmo link

90ee2a0

another wmo link

41412da

put back the links

19d32d3

put back the links

e236b83

Merge remote-tracking branch 'origin/fix-docs' into add-partition-ext…

56b45b9

…ract

github-actions bot added the CI Modifications to autiomation utilities label Dec 15, 2023

juliettelavoie and others added 6 commits December 15, 2023 10:56

Merge remote-tracking branch 'origin/fix-docs' into add-partition-ext…

c50f2de

…ract # Conflicts: # docs/notebooks/4_ensembles.ipynb

import path

81b3a45

try to hide cell

05476ab

merge

a48e05d

merge

b650b79

Merge branch 'main' into add-partition-extract

b70bf53

github-actions bot removed the CI Modifications to autiomation utilities label Dec 15, 2023

juliettelavoie and others added 14 commits December 19, 2023 15:55

build_partition_data

5d1a65d

new doc + fix hidden

463c05a

new test

3ddec06

write the right name

ba029b8

Merge branch 'main' into add-partition-extract

ded0556

[pre-commit.ci] auto fixes from pre-commit.com hooks

4cf76ed

for more information, see https://pre-commit.ci

add a check

6043e53

remove old func

56cc1c2

essayer de diagnostiquer le probleme du build rtd

87dff8f

try again

138ad97

try again str ?

455139b

load le path a la main pour voir

2ece1d7

enleve debuggage

114d880

enleve debuggage open

20d649c

juliettelavoie added 2 commits December 21, 2023 12:12

give up on reusing cat

229698c

path

9d6faab

huard approved these changes Jan 8, 2024

View reviewed changes

RondeauG approved these changes Jan 9, 2024

View reviewed changes

suggestion from review

ae3367d

juliettelavoie added 7 commits January 9, 2024 10:56

Merge branch 'main' into add-partition-extract

04f5b2e

sparse <= 0.14 ?

2fdd40d

Merge remote-tracking branch 'origin/add-partition-extract' into add-…

7dc2949

…partition-extract

dev also?

b7c1324

recipe and meta

39c0e67

toml

926f96b

add reason

df30d75

juliettelavoie merged commit fca2816 into main Jan 10, 2024
18 checks passed

juliettelavoie deleted the add-partition-extract branch January 10, 2024 14:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add get_partition_input #289

add get_partition_input #289

juliettelavoie commented Nov 17, 2023 •

edited

Loading

review-notebook-app bot commented Dec 12, 2023

juliettelavoie commented Dec 21, 2023 •

edited

Loading

huard left a comment

huard Jan 8, 2024

RondeauG left a comment

RondeauG Jan 9, 2024

juliettelavoie commented Jan 9, 2024

	"This tutorial will show how to use the xscen to create the input for [xclim partition functions](https://xclim.readthedocs.io/en/stable/api.html#uncertainty-partitioning)."
	"This tutorial will show how to use xscen to create the input for [xclim partition functions](https://xclim.readthedocs.io/en/stable/api.html#uncertainty-partitioning)."

add get_partition_input #289

add get_partition_input #289

Conversation

juliettelavoie commented Nov 17, 2023 • edited Loading

Pull Request Checklist:

What kind of change does this PR introduce?

Does this PR introduce a breaking change?

Other information:

review-notebook-app bot commented Dec 12, 2023

juliettelavoie commented Dec 21, 2023 • edited Loading

huard left a comment

Choose a reason for hiding this comment

huard Jan 8, 2024

Choose a reason for hiding this comment

RondeauG left a comment

Choose a reason for hiding this comment

RondeauG Jan 9, 2024

Choose a reason for hiding this comment

juliettelavoie commented Jan 9, 2024

juliettelavoie commented Nov 17, 2023 •

edited

Loading

juliettelavoie commented Dec 21, 2023 •

edited

Loading