FEAT-#5836: Introduce 'partial' dtypes cache

[dchigarev]

What do these changes do?

This PR was brought in order to decrease the amount of expensive ._compute_dtypes() calls. This was done by advancing the mechanisms of how we store, use, and update the dtypes cache.

How it was before

Previously, dtypes cache only been able to store "complete" schemas, if we don't know the dtype for at least one column in the dataframe, we set the whole cache to be None.

How it is now

Partially known dtypes

The PR introduced a new class called DtypesDescriptor which is able to store partially known schemas and provides an API to work with them. It's then suggested, that if it's really required to materialize the complete schema, the DtypesDescriptor will only compute dtypes for the columns that we don't have any info about.

There could be several types of partially known schemas:

1. We know the schema for a subset of columns and don't know any other info about the rest of the columns:
   DtypesDescriptor(known_dtypes={"a": ..., "b": ...}, know_all_names=False)
2. We know the schema for a subset of columns and also known names of columns with unknown dtypes:
   DtypesDescriptor(known_dtypes={"a": ..., "b": ...}, cols_with_unknown_dtypes=["c", "d"])
3. We know the schema for a subset of columns and also known a dtype for the rest of the columns:
   DtypesDescriptor(known_dtypes={"a": ..., "b": ...}, remaining_dtype=np.dtype(bool))
4. We only know a common dtype for the whole dataframe:
   DtypesDescriptor(remaining_dtype=np.dtype(bool))
5. ...

How to efficiently use them

It was found out, that there are a lot of cases we don't need to know the complete scheme of a dataframe. For example, we trigger the .dtypes field a lot in our front-end just to verify, whether a dataframe fully constist of numerical columns. It's obvious that we don't need to know the scheme for that, but only a set of dtypes. For that purpose, there was introduced a method called DtypesDescriptor.get_list_of_dtypes() that helped to eliminate ._compute_dtypes() calls in a few places of one of our workload that we're optimizing right now.

There's also a method called DtypesDescriptor.lazy_get(subset) that is able to take a subset of partially known dtypes. This is mainly used in masking. For example:

>>> df.columns # DtypesDescriptor(known_dtypes={"a": int, "b": float}, cols_with_unknown_dtypes=["c", "d"])
["a", "b", "c", "d"]
>>> subset = df[["a", "b"]] # DtypesDescriptor(known_dtypes={"a": int, "b": float}, all_cols_are_known=True)
>>>
>>> df2.columns # DtypesDescriptor(known_dtypes={"a": int, "b": float}, remaining_dtype=float)
["a", "b", "c", "d"]
>>> subset = df2[["b", "c", "d"]] # DtypesDescriptor(known_dtypes={"b": float, "c": float, "d": float}, all_cols_are_known=True)

There's also DtypesDescriptor.concat() method, that is able to merge partially known dtypes, this is mainly used on pd.concat(), df.__setitem__(), and df.insert().

At the current state, the following scenario is now able to work completely without triggering ._compute_dtypes() and also with the complete schema being known at the end (previously, there were several ._compute_dtypes() calls and the schema was still unknown in the result):

def test_get_dummies_case(self):
    with mock.patch.object(PandasDataframe, "_compute_dtypes") as patch:
        df = pd.DataFrame(
            {"items": [1, 2, 3, 4], "b": [3, 3, 4, 4], "c": [1, 0, 0, 1]}
        )
        res = pd.get_dummies(df, columns=["b", "c"])
        cols = [col for col in res.columns if col != "items"]
        res[cols] = res[cols] / res[cols].mean()
        assert res._query_compiler._modin_frame.has_materialized_dtypes
    patch.assert_not_called()

• first commit message and PR title follow format outlined here

  NOTE: If you edit the PR title to match this format, you need to add another commit (even if it's empty) or amend your last commit for the CI job that checks the PR title to pick up the new PR title.

• passes flake8 modin/ asv_bench/benchmarks scripts/doc_checker.py
• passes black --check modin/ asv_bench/benchmarks scripts/doc_checker.py
• signed commit with git commit -s
• Resolves Introduce dtypes cache that can have certain columns to be unknown #5836
• tests added and passing
• module layout described at docs/development/architecture.rst is up-to-date

[dchigarev]

return updated value for convenience

[dchigarev]

_take_2d_positional doesn't apply deferred labels which failed some of the new tests, added this simple workaround until we figure out whether we need lazy_metadata_decorator there or not (#0000 TODO: raise an issue)

[dchigarev]

this method is required to use ModinIndex as a regular index

[dchigarev]

an example of how we can use 'remaining_dtype' functionality, later we should find more places where we can apply this

[dchigarev]

we changed this method in PandasDataframe to return dtypes with an updated parent

[dchigarev]

an example of how we can use get_dtypes_set() functionality, instead of materializing the whole schema, we only request a set of dtypes, there are more places like this in functions that do is_numeric_dtype() checks

[dchigarev]

we now have some dtypes cache quite often, so the old check no longer works correctly, what it wants to check is whether we have materialized dtypes

[dchigarev]

@anmyachev @YarShev @AndreyPavlenko the PR is now ready for review

[anmyachev]

@dchigarev amazing changes! I need more time to finish the review, but it’s already clear that more tests are needed, since there are a lot of them :)

Overall LGTM, but I want to take a closer look at the some implementation details.

[anmyachev]

@dchigarev False positive?

[dchigarev]

I think so, because this code is tested and works properly

[anmyachev]

Why is the "The length of 'columns_order' doesn't match to 'known_dtypes' and 'cols_with_unknown_dtypes'" exception thrown in the constructor when the lengths do not match, but here there is a recalculation?

[dchigarev]

Because if you know columns_order beforehand, you can and IMHO must complete known_dtypes and cols_with_unk... with the information you have at your own. The DtypesDescriptor constructor's parameter matrix is already quite complicated, and I didn't want to bring yet another degree of freedom like "oh, and you also can provide some incomplete argument and we'll infer everything magically from the rest of the arguments". I rather wanted to put as many limitations as possible in order to simplify the constructor's logic and avoid having potentially missed and unprocessed cases because of the bloated parameters matrix.

[anmyachev]

Why are you using set here?

[dchigarev]

just to ignore the order

[anmyachev]

We can ignore it here because we there is the following check: self.columns_order == other.columns_order?

[dchigarev]

nope, consider this example:

dt1 = DtypesDescriptor(cols_with_unknown_dtypes=["a", "b"], columns_order={0: "a", 1: "b"})
dt2 = DtypesDescriptor(cols_with_unknown_dtypes=["b", "a"], columns_order={0: "a", 1: "b"})
dt1.equals(dt2) # should be true

dt1 = DtypesDescriptor(cols_with_unknown_dtypes=["a", "b"], columns_order=None)
dt2 = DtypesDescriptor(cols_with_unknown_dtypes=["b", "a"], columns_order=None)
dt1.equals(dt2) # should be true

[anmyachev]

I see, thanks

[anmyachev]

LGTM!

[dchigarev]

LGTM!

@anmyachev just wanted to bring your attention to #6737, I want this to be merged first, as otherwise, the #6663 changes will carry incorrect dtype through the workload we're optimizing and it will result into perf degradation as it now would perform object-like math rather than float-like math

[anmyachev]

LGTM!

@anmyachev just wanted to bring your attention to #6737, I want this to be merged first, as otherwise, the #6663 changes will carry incorrect dtype through the workload we're optimizing and it will result into perf degradation as it now would perform object-like math rather than float-like math

Merged