From 4c1f0cd538ec5cc0533f288c87760c75f4d31c89 Mon Sep 17 00:00:00 2001 From: Joseph Hamman Date: Tue, 13 Aug 2024 16:06:09 -0700 Subject: [PATCH 1/5] doc(v3): add start of v3 migration page --- docs/index.rst | 1 + docs/installation.rst | 92 ++++++++++++++++++++++++++++++++++++------- 2 files changed, 79 insertions(+), 14 deletions(-) diff --git a/docs/index.rst b/docs/index.rst index 92f9a3df18..cb7b61fa40 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -9,6 +9,7 @@ Zarr-Python :hidden: getting_started + migration tutorial api/index spec diff --git a/docs/installation.rst b/docs/installation.rst index 3d4ac41072..0ae4f7c013 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -1,22 +1,86 @@ -Installation -============ +Zarr-Python 3 Migration Guide +============================= -Zarr depends on NumPy. It is generally best to `install NumPy -`_ first using whatever method is most -appropriate for your operating system and Python distribution. Other dependencies should be -installed automatically if using one of the installation methods below. +Zarr-Python 3 introduces a number of changes to the API, including a number +of significant breaking changes and pending deprecations. -Install Zarr from PyPI:: +This page provides a guide highlighting the most notable changes to help you +migrate your code from Zarr-Python 2.x to Zarr-Python 3.x. - $ pip install zarr +Compatibility target +-------------------- -Alternatively, install Zarr via conda:: +Zarr-Python 3 represents a major refactor of the Zarr-Python codebase. Some of the goals motivating this refactor included: - $ conda install -c conda-forge zarr + - adding support for the V3 specification (alongside the V3 specification) + - cleaning up internal and user facing APIs + - improving performance (particularly in high latency storage environments like cloud object store) -To install the latest development version of Zarr, you can use pip with the -latest GitHub main:: +Though these goals necessitated some breaking changes to the API (hence the major version update), we have tried to maintain +backwards compatibility in the most widely used parts of the API including the `Array` and `Group` classes and the top-level +API (e.g. `zarr.open_array` and `zarr.open_group`). - $ pip install git+https://github.com/zarr-developers/zarr-python.git +Getting ready for 3.0 +--------------------- -To work with Zarr source code in development, see `Contributing `_. \ No newline at end of file +Ahead of the 3.0 release, we suggest projects that depend on Zarr-Python take the following actions: + +1. Pin the supported Zarr-Python version to ``zarr>=2,<3``. This is a best practice and will protect your users from any incompatibilities that may arise during the release of Zarr-Python 3.0. +2. Limit your imports from the Zarr-Python package. Most of the primary API ``zarr.*`` will be compatible in 3.0. However, the following breaking API changes are planned: + + - ``numcodecs.*`` will no longer be available in ``zarr.*``. (Suggested action: transition to importing codecs from ``numcodecs`` directly.) + - The ``zarr.v3_api_available`` feature flag is being removed. (Suggested action: this experimental feature was deprecated in v2.18.) + - The following internal modules are being removed or significant changed: + + - ``zarr.attrs`` + - ``zarr.codecs`` + - ``zarr.context`` + - ``zarr.core`` + - ``zarr.hierarchy`` + - ``zarr.indexing`` + - ``zarr.meta`` + - ``zarr.meta_v1`` + - ``zarr.storage`` + - ``zarr.sync`` + - ``zarr.types`` + - ``zarr.util`` + +Continue using Zarr-Python 2 +---------------------------- + +Zarr-Python 2.x is still available, though we recommend migrating to Zarr-Python 3 for its improvements and new features. + +If you need to use the latest Zarr-Python 2 release, you can install it with: + + $ pip install "zarr==2.*" + + +Migration Guide +--------------- + +The following sections provide details on the most important changes in Zarr-Python 3. + +Changes to the Array class +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +TODO + +Changes to the Group class +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +TODO + +Changes to the Store class +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some of the biggest changes in Zarr-Python 3 are found in the `Store` class. The most notable changes to the Store API are: + +1. Dropped the ``MutableMapping`` base class in favor of a custom abstract base class (``zarr.abc.store.Store``). +2. Switched to a primarily Async interface. + +TODO + +Miscellaneous changes +~~~~~~~~~~~~~~~~~~~~~ + +TODO From d4192c15fbb58f3ee98693af11f2dcef886304bd Mon Sep 17 00:00:00 2001 From: Joseph Hamman Date: Mon, 19 Aug 2024 16:31:07 -0700 Subject: [PATCH 2/5] add missing doc --- docs/installation.rst | 92 +++++++------------------------------------ docs/migration.rst | 86 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 100 insertions(+), 78 deletions(-) create mode 100644 docs/migration.rst diff --git a/docs/installation.rst b/docs/installation.rst index 0ae4f7c013..ca2f7a5f1a 100644 --- a/docs/installation.rst +++ b/docs/installation.rst @@ -1,86 +1,22 @@ -Zarr-Python 3 Migration Guide -============================= +Installation +============ -Zarr-Python 3 introduces a number of changes to the API, including a number -of significant breaking changes and pending deprecations. +Zarr depends on NumPy. It is generally best to `install NumPy +`_ first using whatever method is most +appropriate for your operating system and Python distribution. Other dependencies should be +installed automatically if using one of the installation methods below. -This page provides a guide highlighting the most notable changes to help you -migrate your code from Zarr-Python 2.x to Zarr-Python 3.x. +Install Zarr from PyPI:: -Compatibility target --------------------- + $ pip install zarr -Zarr-Python 3 represents a major refactor of the Zarr-Python codebase. Some of the goals motivating this refactor included: +Alternatively, install Zarr via conda:: - - adding support for the V3 specification (alongside the V3 specification) - - cleaning up internal and user facing APIs - - improving performance (particularly in high latency storage environments like cloud object store) + $ conda install -c conda-forge zarr -Though these goals necessitated some breaking changes to the API (hence the major version update), we have tried to maintain -backwards compatibility in the most widely used parts of the API including the `Array` and `Group` classes and the top-level -API (e.g. `zarr.open_array` and `zarr.open_group`). +To install the latest development version of Zarr, you can use pip with the +latest GitHub main:: -Getting ready for 3.0 ---------------------- + $ pip install git+https://github.com/zarr-developers/zarr-python.git -Ahead of the 3.0 release, we suggest projects that depend on Zarr-Python take the following actions: - -1. Pin the supported Zarr-Python version to ``zarr>=2,<3``. This is a best practice and will protect your users from any incompatibilities that may arise during the release of Zarr-Python 3.0. -2. Limit your imports from the Zarr-Python package. Most of the primary API ``zarr.*`` will be compatible in 3.0. However, the following breaking API changes are planned: - - - ``numcodecs.*`` will no longer be available in ``zarr.*``. (Suggested action: transition to importing codecs from ``numcodecs`` directly.) - - The ``zarr.v3_api_available`` feature flag is being removed. (Suggested action: this experimental feature was deprecated in v2.18.) - - The following internal modules are being removed or significant changed: - - - ``zarr.attrs`` - - ``zarr.codecs`` - - ``zarr.context`` - - ``zarr.core`` - - ``zarr.hierarchy`` - - ``zarr.indexing`` - - ``zarr.meta`` - - ``zarr.meta_v1`` - - ``zarr.storage`` - - ``zarr.sync`` - - ``zarr.types`` - - ``zarr.util`` - -Continue using Zarr-Python 2 ----------------------------- - -Zarr-Python 2.x is still available, though we recommend migrating to Zarr-Python 3 for its improvements and new features. - -If you need to use the latest Zarr-Python 2 release, you can install it with: - - $ pip install "zarr==2.*" - - -Migration Guide ---------------- - -The following sections provide details on the most important changes in Zarr-Python 3. - -Changes to the Array class -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -TODO - -Changes to the Group class -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -TODO - -Changes to the Store class -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Some of the biggest changes in Zarr-Python 3 are found in the `Store` class. The most notable changes to the Store API are: - -1. Dropped the ``MutableMapping`` base class in favor of a custom abstract base class (``zarr.abc.store.Store``). -2. Switched to a primarily Async interface. - -TODO - -Miscellaneous changes -~~~~~~~~~~~~~~~~~~~~~ - -TODO +To work with Zarr source code in development, see `Contributing `_. diff --git a/docs/migration.rst b/docs/migration.rst new file mode 100644 index 0000000000..0ae4f7c013 --- /dev/null +++ b/docs/migration.rst @@ -0,0 +1,86 @@ +Zarr-Python 3 Migration Guide +============================= + +Zarr-Python 3 introduces a number of changes to the API, including a number +of significant breaking changes and pending deprecations. + +This page provides a guide highlighting the most notable changes to help you +migrate your code from Zarr-Python 2.x to Zarr-Python 3.x. + +Compatibility target +-------------------- + +Zarr-Python 3 represents a major refactor of the Zarr-Python codebase. Some of the goals motivating this refactor included: + + - adding support for the V3 specification (alongside the V3 specification) + - cleaning up internal and user facing APIs + - improving performance (particularly in high latency storage environments like cloud object store) + +Though these goals necessitated some breaking changes to the API (hence the major version update), we have tried to maintain +backwards compatibility in the most widely used parts of the API including the `Array` and `Group` classes and the top-level +API (e.g. `zarr.open_array` and `zarr.open_group`). + +Getting ready for 3.0 +--------------------- + +Ahead of the 3.0 release, we suggest projects that depend on Zarr-Python take the following actions: + +1. Pin the supported Zarr-Python version to ``zarr>=2,<3``. This is a best practice and will protect your users from any incompatibilities that may arise during the release of Zarr-Python 3.0. +2. Limit your imports from the Zarr-Python package. Most of the primary API ``zarr.*`` will be compatible in 3.0. However, the following breaking API changes are planned: + + - ``numcodecs.*`` will no longer be available in ``zarr.*``. (Suggested action: transition to importing codecs from ``numcodecs`` directly.) + - The ``zarr.v3_api_available`` feature flag is being removed. (Suggested action: this experimental feature was deprecated in v2.18.) + - The following internal modules are being removed or significant changed: + + - ``zarr.attrs`` + - ``zarr.codecs`` + - ``zarr.context`` + - ``zarr.core`` + - ``zarr.hierarchy`` + - ``zarr.indexing`` + - ``zarr.meta`` + - ``zarr.meta_v1`` + - ``zarr.storage`` + - ``zarr.sync`` + - ``zarr.types`` + - ``zarr.util`` + +Continue using Zarr-Python 2 +---------------------------- + +Zarr-Python 2.x is still available, though we recommend migrating to Zarr-Python 3 for its improvements and new features. + +If you need to use the latest Zarr-Python 2 release, you can install it with: + + $ pip install "zarr==2.*" + + +Migration Guide +--------------- + +The following sections provide details on the most important changes in Zarr-Python 3. + +Changes to the Array class +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +TODO + +Changes to the Group class +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +TODO + +Changes to the Store class +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Some of the biggest changes in Zarr-Python 3 are found in the `Store` class. The most notable changes to the Store API are: + +1. Dropped the ``MutableMapping`` base class in favor of a custom abstract base class (``zarr.abc.store.Store``). +2. Switched to a primarily Async interface. + +TODO + +Miscellaneous changes +~~~~~~~~~~~~~~~~~~~~~ + +TODO From 3dbc86cc51784b7ca038a868c20a3715d3152d1d Mon Sep 17 00:00:00 2001 From: Joseph Hamman Date: Mon, 19 Aug 2024 16:36:45 -0700 Subject: [PATCH 3/5] more details --- docs/migration.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/migration.rst b/docs/migration.rst index 0ae4f7c013..120d3f821e 100644 --- a/docs/migration.rst +++ b/docs/migration.rst @@ -18,7 +18,8 @@ Zarr-Python 3 represents a major refactor of the Zarr-Python codebase. Some of t Though these goals necessitated some breaking changes to the API (hence the major version update), we have tried to maintain backwards compatibility in the most widely used parts of the API including the `Array` and `Group` classes and the top-level -API (e.g. `zarr.open_array` and `zarr.open_group`). +API (e.g. `zarr.open_array` and `zarr.open_group`). It is worth noting that we significantly evolved the internal data model, +moving away from a model that was tightly coupled to the v2 specification, and to a more generic representation of Zarr objects. Getting ready for 3.0 --------------------- @@ -44,11 +45,13 @@ Ahead of the 3.0 release, we suggest projects that depend on Zarr-Python take th - ``zarr.sync`` - ``zarr.types`` - ``zarr.util`` + - ``zarr.n5`` Continue using Zarr-Python 2 ---------------------------- Zarr-Python 2.x is still available, though we recommend migrating to Zarr-Python 3 for its improvements and new features. +Security and bug fixes will be made to the 2.x series for at least 6 months following the first Zarr-Python 3 release. If you need to use the latest Zarr-Python 2 release, you can install it with: @@ -63,12 +66,12 @@ The following sections provide details on the most important changes in Zarr-Pyt Changes to the Array class ~~~~~~~~~~~~~~~~~~~~~~~~~~ -TODO +1. Disallow direct construction Changes to the Group class ~~~~~~~~~~~~~~~~~~~~~~~~~~ -TODO +1. Disallow direct construction Changes to the Store class ~~~~~~~~~~~~~~~~~~~~~~~~~~ From 5469a614959953b66da861caaf24b2dad69ae9f9 Mon Sep 17 00:00:00 2001 From: Joe Hamman Date: Fri, 13 Sep 2024 15:20:07 -0700 Subject: [PATCH 4/5] Apply suggestions from code review Co-authored-by: David Stansby --- docs/migration.rst | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/migration.rst b/docs/migration.rst index 120d3f821e..3f7cfb0b88 100644 --- a/docs/migration.rst +++ b/docs/migration.rst @@ -5,21 +5,20 @@ Zarr-Python 3 introduces a number of changes to the API, including a number of significant breaking changes and pending deprecations. This page provides a guide highlighting the most notable changes to help you -migrate your code from Zarr-Python 2.x to Zarr-Python 3.x. +migrate your code from version 2 to version 3. Compatibility target -------------------- Zarr-Python 3 represents a major refactor of the Zarr-Python codebase. Some of the goals motivating this refactor included: - - adding support for the V3 specification (alongside the V3 specification) + - adding support for the Zarr V3 specification (alongside the Zarr V2 specification) - cleaning up internal and user facing APIs - improving performance (particularly in high latency storage environments like cloud object store) -Though these goals necessitated some breaking changes to the API (hence the major version update), we have tried to maintain +These goals necessitated some breaking changes to the API (hence the major version update), but we have tried to maintain backwards compatibility in the most widely used parts of the API including the `Array` and `Group` classes and the top-level -API (e.g. `zarr.open_array` and `zarr.open_group`). It is worth noting that we significantly evolved the internal data model, -moving away from a model that was tightly coupled to the v2 specification, and to a more generic representation of Zarr objects. +API (e.g. `zarr.open_array` and `zarr.open_group`). Getting ready for 3.0 --------------------- @@ -30,8 +29,8 @@ Ahead of the 3.0 release, we suggest projects that depend on Zarr-Python take th 2. Limit your imports from the Zarr-Python package. Most of the primary API ``zarr.*`` will be compatible in 3.0. However, the following breaking API changes are planned: - ``numcodecs.*`` will no longer be available in ``zarr.*``. (Suggested action: transition to importing codecs from ``numcodecs`` directly.) - - The ``zarr.v3_api_available`` feature flag is being removed. (Suggested action: this experimental feature was deprecated in v2.18.) - - The following internal modules are being removed or significant changed: + - The ``zarr.v3_api_available`` feature flag is being removed. In zarr-python v3 the v3 API is always available, so you shouldn't need to use this flag. + - The following internal modules are being removed or significantly changed. If your application relies on imports from any of the below modules, you will need to either a) modify your application to no longer rely on these imports or b) vendor the parts of the specific modules that you need. - ``zarr.attrs`` - ``zarr.codecs`` @@ -47,6 +46,8 @@ Ahead of the 3.0 release, we suggest projects that depend on Zarr-Python take th - ``zarr.util`` - ``zarr.n5`` +3. Test that your package works with v3. You can start testing against version 3 now (pre-releases are being published to PyPI weekly). +4. Update the pin to zarr >=3 Continue using Zarr-Python 2 ---------------------------- From fa14ca7e68a2810b4d4f2e361b6361df6ff4d40c Mon Sep 17 00:00:00 2001 From: Joe Hamman Date: Fri, 13 Sep 2024 15:20:29 -0700 Subject: [PATCH 5/5] Update docs/migration.rst --- docs/migration.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/migration.rst b/docs/migration.rst index 3f7cfb0b88..51a82bb851 100644 --- a/docs/migration.rst +++ b/docs/migration.rst @@ -48,6 +48,7 @@ Ahead of the 3.0 release, we suggest projects that depend on Zarr-Python take th 3. Test that your package works with v3. You can start testing against version 3 now (pre-releases are being published to PyPI weekly). 4. Update the pin to zarr >=3 + Continue using Zarr-Python 2 ----------------------------