Universal API implementations for Micro-Manager OME-TIFF and NDTiff

[ziw-liu]

Implements universal API (UAPI) for MMStack (Micro-Manager OME-TIFF) and NDTiff (written by MM and pycro-manager). This PR contains many breaking changes and is geared towards the v0.2.0 release.

Unresolved issues

• UAPI for OME-NGFF
• Root path of BaseFOV #157 (@JoOkuma)

User-facing changes

API

• UAPI entry point (Universal entry points #40): rename read_micromanager to read_images
  • Removed extract_data and format arguments
  • Suggestions to function name are welcome!
  • TODO: include clear control?
• UAPI for MM OME-TIFF
  • Replace multipagetiff.MicromanagerOmeTiffReader with mmstack.MMStack
    • Closes The MicromanagerOmeTiffReader needs better definitions or documentation for position attributes #152 (@ieivanov)
    • Closes Refactor _set_mm_meta in BaseReader and its children #85 (@JoOkuma)
  • When the channel names from metadata is incomplete (e.g. multi-camera mode on the Dragonfly microscope), the name list will be padded with generic names, e.g. [..., "Channel2", "Channel3", ...]. Note that this will be incorrect if the partially available channel names do not start with the first channel or if they are not contiguous. A warning will be logged and the user is responsible to correct this downstream.
• UAPI for NDTiff
  • Replace ndtiff.NDTiffReader with ndtiff.NDTiffDataset
  • When channel names are not available from metadata, fallback to ['Channel0', 'Channel1', ...] instead of ['0', '1', ...]
• OME-TIFF and NDTiff FOV objects now have a frame_metadata object as proposed in Per-frame metadata in UAPI #155
  • Closes The MicromanagerOmeTiffReader needs methods to retrieve the image plane metadata #151 (@ieivanov)
• Non-UAPI readers (MM single-page tiff, v0.1 OME-Zarr, uPTI) are moved into the _deprecated namespace
  • Only accessible from specific API (importing corresponding reader classes)
  • Testing is still maintained and there is no immediate plan to remove them
• When ZYX scaling information is not available from OME-TIFF metadata, the reader will fall back to 1.0 and log a warning.
  • Closes iohub info fails on MM datasets without xy_pixel_size #180 (@talonchandler)
  • Replaces Change exception to warning if xy_scale_size metadata is absent #182 (@talonchandler)
• Renamed convert.TIFFConverter.run() to convert.TIFFConverter.__call__()

CLI

• Removed support for converting single-page TIFFs
• Removed arguments from the convert command
  • --format is always auto-inferred
  • --scale-voxels is always TRUE
  • --check-image is always FALSE
• The chunk size argument --chunks now works for both NDTiff and OME-TIFF
  • The default chunk size is ZYX for all input formats
• Avoid using the root logger and set the default logging level to INFO (better logging #188)
  • The logging level can be changed with an environment variable, e.g. IOHUB_LOG_LEVEL=DEBUG

Documentation

• Updated sphinx theme
• TODO: add version switcher for old releases

Technical changes

• Use tifffile's TiffZarrStore to acheive lazy loading of OME-TIFF. The custom index map parsing is removed.
  • Closes Improve TIFF I/O stack #53
• Use xarray to provide named axes in the TCZYX order. (The xdata attribute gives access to the underlying xarray object.)
• Bumps tifffile and zarr-python to latest
  • Closes unpin zarr versions #207 (@JoOkuma)
  • Closes Single-page TIFF converter regression? #164
  • Replaces pin tifffile to a newer version to allow unpinning of ome-zarr. #208 (@ilan-francisco)
• Reader testing is now done for all the applicable files
  • Replaced session-scoped fixtures in conftest.py with globals since parametrization happens before fixture is available

Notes

Many issues are linked to be closed by this PR.
Reviewers: please check the boxes and leave a sign-off when an issue has been verified to be fixed.

[ziw-liu]

@JoOkuma Not sure if this is going to work for the DaXi data since I don't have 1024x2048x2048 volumes to test with...
Also if you don't like the Dask conversion, the TiffZarrStore (in acquisition axes ordering) is available through MMStack._store.

[JoOkuma]

@ziw-liu, thank you. I'll give it a try soon

[mattersoflight]

@ziw-liu fantastic work across all the layers of iohub. @ieivanov great suggestions!
It looks like some of the open issues (#71 , #207 , #208 ) are quick to handle. Please test/fix the code if the original authors are busy and close them.

I suggest merging when these features are implemented and tested with real data:

• iohub convert can chunk by z (looks like that was done in another PR).
• Unified API is documented on README so that power users/reviewers begin to switch and can provide feedback aligned with future development.

I think the following can be future/smaller PRs that quickly follow this:

• Proper metadata parsing across the formats. How we handle metadata depends on how we will consume metadata - in the viewer, through API, etc. One PR per format would make sense to iterate through the metadata handling systematically.
• User friendly examples.

I don't have input on when it is best to rename methods. Since many users are now finding this PR valuable and will provide input as they use the code, it is smart not to delay the merge.

[ziw-liu]

@ieivanov Is there any blocking issues left before we are ready to merge this?

[ieivanov]

One question / request before I approve - when are the docs rebuilt? It looks like only on a push to main? It will be very helpful to have docs built for this PR (or the ometiff-uapi branch) so that users can have easy access to the updated API.

[ziw-liu]

One question / request before I approve - when are the docs rebuilt? It looks like only on a push to main? It will be very helpful to have docs built for this PR (or the ometiff-uapi branch) so that users can have easy access to the updated API.

Currently it is only built when pushing to main. I can attempt to also build for this the unified-api branch, but we won't know if it is going to work (before we actually merge this).

[ziw-liu]

@ieivanov I propose that we handle the docs preview in a separate PR so we can easily revert the change after the transition.

[ieivanov]

Let's get this PR merged. A few outstanding non-blocking issues from my point of view based on our discussion so far:

• I suggest renaming classes as in Universal API implementations for Micro-Manager OME-TIFF and NDTiff #185 (comment). This does not introduce any use-facing changes, but I think would improve the readability of the code.
• build docs preview showing the new UAPI, to be resolved in a different PR

[ieivanov]

I think there is a bug / confusion around the root property of reader classes. BaseFOV defines a root property and BaseFOVMapping does not. MicroManagerFOV then defines

  @property
  def root(self) -> Path:
      return self.parent.root

but its parent MicroManagerFOVMapping does not have a root property. Later on, both MMStack and NDTiffDataset define a root property, even though it's not required in the base MicroManagerFOVMapping class.

Should root be a property of a FOV or a Stack/Dataset?

I think there may be a similar issue with zyx_scale and t_scale - should these be properties of the FOV or Stack/Dataset? Would we allow a Stack/Dataset to contain FOVs with different scale?

[JoOkuma]

stack could be replaced with dataset, if you prefer. One more iteration could be:

I like @ieivanov's suggestion; I think dataset would be more appropriate because stack is a "well-defined" data structure, which we do not use for these classes.

[JoOkuma]

Once again, I would like to thank @ziw-liu for this PR. It's a huge change and will improve our implementation of downstream image processing/analysis routines.

It closes the Zarr and file issues we raised before.

My concerns are:

• The primary usage of xdata on the new FOV classes, which deviates from the common API. From what I chatted with @ziw-liu, this is a requirement for the TIFF data (so it has array-like behavior? I don't remember quite well).
  I'm worried some routines start depending on it and, therefore, are not universal, which is the main goal of this PR.
  This method could be added to BaseFOV so every class must implement its own xdata method; the downside is BaseFOV starts deviating slightly from being an ArrayLike object.
• The convert function being TIFF specific. The original vision of the universal API was that the conversion function would be BaseFOVMapping -> BaseFOVMapping --- despite, in practice, the output type being a subset of all supported types because some are read-only.
  This leads me to think that the API isn't universal yet.
  I understand that MicroManager-specific features are not available in other classes, but I think these could be wrapped into existing BaseFOVMapping methods; if that's not the case, the BaseFOVMapping could be improved.

[ziw-liu]

Should root be a property of a FOV or a Stack/Dataset?

I think there may be a similar issue with zyx_scale and t_scale - should these be properties of the FOV or Stack/Dataset? Would we allow a Stack/Dataset to contain FOVs with different scale?

For MM formats, both FOV and Mapping classes should have the root and zyx_scale attributes and they should be the same value (I added them to MicroManagerFOVMapping). Requiring these attributes for BaseFOV is to make it work in the same way for single-FOV formats (e.g. CC).

For NDTIFF, I couldn't find a way to get the intended time interval, so the t_scale implementation is specific to MMOmeTiffFOV for now.

For scaling, I think we should assume homogeneous mappings but allow it to be used for expressing heterogeneous datasets.

[ziw-liu]

@JoOkuma Can #85 be closed by this?

[ziw-liu]

The primary usage of xdata on the new FOV classes, which deviates from the common API. From what I chatted with @ziw-liu, this is a requirement for the TIFF data (so it has array-like behavior? I don't remember quite well).
I'm worried some routines start depending on it and, therefore, are not universal, which is the main goal of this PR.
This method could be added to BaseFOV so every class must implement its own xdata method; the downside is BaseFOV starts deviating slightly from being an ArrayLike object.

xarray.DataArray is also ArrayLike. In the case of TIFF readers it is only to label the axes of the underlying dask.Array. I think the spirit of UAPI was that one can call napari.view_image with any BaseFOV child.
Edit: IIRC we decided that the full array interface is too broad to implement.

[JoOkuma]

@JoOkuma Can #85 be closed by this?

@ziw-liu , I think so.

[JoOkuma]

The primary usage of xdata on the new FOV classes, which deviates from the common API. From what I chatted with @ziw-liu, this is a requirement for the TIFF data (so it has array-like behavior? I don't remember quite well).
I'm worried some routines start depending on it and, therefore, are not universal, which is the main goal of this PR.
This method could be added to BaseFOV so every class must implement its own xdata method; the downside is BaseFOV starts deviating slightly from being an ArrayLike object.

xarray.DataArray is also ArrayLike. In the case of TIFF readers it is only to label the axes of the underlying dask.Array. I think the spirit of UAPI was that one can call napari.view_image with any BaseFOV child. Edit: IIRC we decided that the full array interface is too broad to implement.

Ok, now that I applied closer attention I understand that you're mainly using xdata inside the __getitem__ methods.
I was a bit concerned because of

to_zarr(fov.xdata.data.rechunk(self.chunks), zarr_img)

But there isn't an easy way around it.

@ziw-liu , I'm very happy with the PR, and for me it's ready to merge and remaining tasks could be converted to issues.

[ziw-liu]

@ieivanov @JoOkuma @talonchandler Thanks for the reviews!