chore(datasets): Remove tracking datasets which are used in Kedro Viz Experiment Tracking

[ravi-kumar-pilla]

Description

Resolves kedro-org/kedro#4370

NOTE: This should be merged after kedro-datasets 6.0.0 release

Development notes

• Remove docs references
• Remove json_dataset and metrics_dataset
• Remove pytests related to tracking datasets

Checklist

• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Updated jsonschema/kedro-catalog-X.XX.json if necessary
• Added a description of this change in the relevant RELEASE.md file
• Added tests to cover my changes
• Received approvals from at least half of the TSC (required for adding a new, non-experimental dataset)

[ankatiyar]

👋🏾

[DimedS]

Thanks, @ravi-kumar-pilla !

[ravi-kumar-pilla]

Hi @ankatiyar and @DimedS ,

The docs build fails as dask deprecated the usage of dask.dataframe and shifted to dask-expr. As a temporary fix in this PR, I have updated the docstring to stable doc link. I will create another ticket if we decide on migrating to dask-expr. Let me know your thoughts.

Thank you

[ankatiyar]

@ravi-kumar-pilla The docs fix seems fine for now, if the dataset itself is working fine. We can create an issue for the migration!

[DimedS]

@ravi-kumar-pilla The docs fix seems fine for now, if the dataset itself is working fine. We can create an issue for the migration!

I agree, thanks @ravi-kumar-pilla and @ankatiyar !

[ravi-kumar-pilla]

Hi @ankatiyar and @DimedS ,

Thanks for the response. After further reading, I found that Dask-Expr is a new backend for Dask DataFrame that provides query optimization and other performance improvements. Starting with Dask version 2024.3, it became the default backend.

I don't think there is a need for migration from our side unless we plan to opt out of the optimization by setting the below config -

import dask
dask.config.set({'dataframe.query-planning': False})

Note

The above config is also set to be removed as it had some issues mentioned here

As per the docs, they actually updated the docs link as mentioned here. So the fix in this PR is suffice.

Thank you