Dask: KMeans

[noahnovsak]

Add dask_ml.cluster.KMeans as an alternative to sklearn.cluster.KMeans for dask arrays.

Probably worth mentioning, dask_ml adds the "k-means||" init option but defaults to the sklearn implementation in the case of "k-means++" or "random" initialization. So using "k-means||" is what enables working with larger datasets at all. However, smaller datasets are still processed much faster just using sklearn directly as there seems to be a lot of overhead with dask_ml.

[markotoplak]

This is a good find. I remember you telling me about it, but then I forgot. I do not have a good solution for now, but we would need to ensure (and test) at the minimum that array_equal does not load the whole data set into memory.

[noahnovsak]

Turns out np.allclose works fine on dask. Comparing an 8GB array against itself took a little over two seconds (this is worst case scenario, runing in an ipython shell that doesn't play well with dask) and memory usage stayed very much within reasonable limits.

[markotoplak]

Please add dask-ml to requirements (that means tox.ini as the oldest version specification, requirements-?.txt, and meta.yaml).

Next, this does not actually use context settings: they would always need a context handler (in this case it should detect whether you have a dask table or not), then they need a openContext and closeContext calls.

Maybe we do not need them here and they were a bad idea I had... But what needs to work is the following:

• use kmeans with ordinary data, select any option, lets call it O
• change input to dask data, select anything
• change back to ordinary data, the option O needs to be selected

From the code I'd guess this does not work at least for some possible O. :)

And then we have to remember that we still need settings migration.

[janezd]

1. In Sample Data we have an checkbox Stratify (if possible). Is this an option here? Combos (I would actually prefer radio buttons, but this may be my current temporary preference) and the one that is not always applicable would indicate what it defaults to. At least in a tooltip.

   Option "O" would remain chosen, but a warning would indicate that it was not used.

Another thing: this is only available for Dask, so it will be off for majority of users. If so, I would disable it when inapplicable. If chosen, I would keep it chosen (but disabled!), and have a warning etc. So the user may manually choose another option, or keep the disabled one, which won't work anyway.

An alternative to above would be to hide the option if it is unavailable and not chosen.

[codecov]

Codecov Report

Merging #6277 (ace9abb) into dask (2f0bec2) will increase coverage by 0.00%.
The diff coverage is 95.23%.

Additional details and impacted files

@@           Coverage Diff           @@
##             dask    #6277   +/-   ##
=======================================
  Coverage   87.64%   87.65%           
=======================================
  Files         322      322           
  Lines       69601    69637   +36     
=======================================
+ Hits        61002    61040   +38     
+ Misses       8599     8597    -2     

[markotoplak]

This change, if we go with it, also requires settings migration. Just leaving it here as a note.

[markotoplak]

Thanks. We need at least some basic tests for this.