Skip to content

Commit

Permalink
DOCS-#6658: Add a note how to enable object spilling in a multi-node …
Browse files Browse the repository at this point in the history
…Ray cluster

Signed-off-by: Anatoly Myachev <[email protected]>
  • Loading branch information
anmyachev committed Oct 17, 2023
1 parent 21518c4 commit a1a48a0
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion docs/getting_started/why_modin/out_of_core.rst
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
Out-of-memory data with Modin
=============================

.. note::
.. note::
| *Estimated Reading Time: 10 minutes*
When using pandas, you might run into a memory error if you are working with large datasets that cannot fit in memory or perform certain memory-intensive operations (e.g., joins).

Modin solves this problem by spilling over to disk, in other words, it uses your disk as an overflow for memory so that you can work with datasets that are too large to fit in memory. By default, Modin leverages out-of-core methods to handle datasets that don't fit in memory for both Ray and Dask engines.

.. note::
Object spilling is disabled in a multi-node Ray cluster by default. To enable object spilling
use `Ray's instruction https://docs.ray.io/en/latest/ray-core/objects/object-spilling.html#cluster-mode`.


Motivating Example: Memory error with pandas
--------------------------------------------
Expand Down

0 comments on commit a1a48a0

Please sign in to comment.