Add blogpost on flox heuristics #695
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
Co-authored-by: Anderson Banihirwe <[email protected]>
| @@ -0,0 +1,157 @@ | |||
| --- | |||
| title: 'flox: GroupBy, now with smarts!' | |||
| date: '2024-05-31' | |||
There was a problem hiding this comment.
| date: '2024-05-31' | |
| date: '2024-08-05' |
There was a problem hiding this comment.
or whenever we decide to merge.
|
|
||
| Unlike [Jaccard similarity](https://en.wikipedia.org/wiki/Jaccard_index), _containment_ [isn't skewed](http://ekzhu.com/datasketch/lshensemble.html) when one of the sets is much larger than the other. | ||
|
|
||
| The steps are as follows: |
There was a problem hiding this comment.
Is this bit clear and useful? Should I take it out? Or expand it?
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
Nice! Really incredible work you have done here @dcherian
I left a few comments: some nits, some minor suggestions on structure/content.
|
|
||
| ## Avoiding catastrophe | ||
|
|
||
| Thus `flox` quickly grew two new modes of computing the groupby reduction. |
There was a problem hiding this comment.
I would move this sentence down to after you have gone through your two realizations. Otherwise it's confusing when you jump into
First,
method="blockwise"
| Notice how much cleaner the graph is in this image: | ||
|  | ||
| See our [previous blog post](https://xarray.dev/blog/flox) or the [docs](https://flox.readthedocs.io/en/latest/implementation.html#method-map-reduce) for more. |
There was a problem hiding this comment.
It's a little confusing that these two images are not for identical problems - the inputs are different.
| ```python | ||
| mean_mapreduce = ds.groupby("time.month").mean(method="map-reduce") | ||
| mean_cohorts = ds.groupby("time.month").mean() # this is auto-detected! | ||
| ``` |
There was a problem hiding this comment.
| ```python | |
| mean_mapreduce = ds.groupby("time.month").mean(method="map-reduce") | |
| mean_cohorts = ds.groupby("time.month").mean() # this is auto-detected! | |
| ``` | |
| ```python | |
| mean_mapreduce = ds.groupby("time.month").mean(method="map-reduce") # mapreduce is a suboptimal manual choice here | |
| mean_cohorts = ds.groupby("time.month").mean() # cohorts is a better choice - auto-detected! |
| Using the algorithm described below, flox will **automatically** set | ||
| `method="cohorts"` for this dataset unless specified, yielding a 5X decrease in | ||
| memory used and a 2X increase in runtime. Read on to figure out how! | ||
| Read on to figure out how! |
There was a problem hiding this comment.
| Read on to figure out how! |
(duplication)
|
|
||
| _Note that the improvements here are strongly dependent on the details | ||
| of the grouping variable, the chunksize, and even dask's scheduler. In fact, writing this post | ||
| prompted the discovery of a bug in dask's scheduler that should substantially improve the "map-reduce" |
|
|
||
| The improvements described here are strongly dependent on the details | ||
| of the grouping variable, the chunksize, and even dask's scheduler. | ||
| In fact, writing this post prompted the discovery of a bug in dask's scheduler |
|
|
||
| Importantly this inference is fast — [~250ms for the US county](https://flox.readthedocs.io/en/latest/implementation.html#example-spatial-grouping) GroupBy problem in our [previous post](https://xarray.dev/blog/flox) where approximately 3000 groups are distributed over 2500 chunks; and ~1.25s for grouping by US watersheds ~87000 groups across 640 chunks. | ||
|
|
||
| ## What's next? |
There was a problem hiding this comment.
Your conclusion is a bit rambly, and I think some of the content (especially call-outs to any other groupby nerds who can help) could be in a footnote.
I think you should end with a succinct one-line conclusion that emphasises that most of this complexity can now be safely ignored by users.
| Using the algorithm described below, flox will **automatically** set | ||
| `method="cohorts"` for this dataset unless specified, yielding a 5X decrease in | ||
| memory used and a 2X increase in runtime. Read on to figure out how! | ||
| Read on to figure out how! |
There was a problem hiding this comment.
A significant number of readers will think it's cool that this is now automatic, but not really care about the specifics of how you did it. That section below is also very dense. I think if there is any other information that you want readers to take away from this point onwards (e.g. that Groupers exist) you should telegraph it, and explicitly give them the option to skip forwards to a different subheading guilt-free.
|
Thanks for the review @TomNicholas . I'm waiting for some tests on the new dask release before revising: https://docs.dask.org/en/stable/changelog.html#improve-scheduling-efficiency-for-xarray-groupby-reduce-patterns |
https://xarray-dev-git-flox-smart-xarray.vercel.app/blog/flox-smart