Reading/Writing DM5 files

[CSSFrancis]

Curious of peoples thoughts on trying to support reading/writing of .dm5 files. The format is pretty much identical to the .dm4 and .dm3 file format except for split into a hierarchical data format. It doesn't seem too hard to write a (minimum) dm5 file that can be read. This would be a good way for people to check/compare data and also good for interoperability.

@jlaehne do you have any example datasets? That might be helpful. I have some 4D STEM datasets but any other data type would be nice to help identify how the guide image should be created.

Description of the change

• Read 4D STEM
• Write 4D STEM
• Read In situ TEM
• Write In Situ TEM
• Read EDS
• Write EDS
• Read EELS
• Write EELS
• update docstring (if appropriate),
• update user guide (if appropriate),
• add a changelog entry in the upcoming_changes folder (see upcoming_changes/README.rst),
• Check formatting of the changelog entry (and eventual user guide changes) in the docs/readthedocs.org:rosettasciio build of this PR (link in github checks)
• add tests,
• ready for review.

Any suggestions might be helpful here. I think I mostly have a good handle on the file format but specifically I'm not sure how important the Meta Data tags are in identifying the "Type" of dataset.

[codecov]

Codecov Report

Attention: Patch coverage is 88.93130% with 29 lines in your changes missing coverage. Please review.

Project coverage is 87.70%. Comparing base (d54f7d1) to head (06ea314).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
rsciio/dm5/_api.py	88.75%	15 Missing and 14 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #315      +/-   ##
==========================================
+ Coverage   87.67%   87.70%   +0.02%     
==========================================
  Files          83       85       +2     
  Lines       11149    11411     +262     
  Branches     2414     2464      +50     
==========================================
+ Hits         9775    10008     +233     
- Misses        860      875      +15     
- Partials      514      528      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jlaehne]

@jlaehne do you have any example datasets? That might be helpful. I have some 4D STEM datasets but any other data type would be nice to help identify how the guide image should be created.

Unfortunately our lab setup has some old interfaces so that we are stuck on a DM 2.72 version. So I can not directly produce sample data sets. If the latest free DM 3.x can write .dm5 files, which I have not looked in to, we could of course convert existing test data sets from .dm4 to .dm5 (otherwise I know one or two labs that might be able to produce such data sets for CL though).

If Gatan is already making the move to an hdf5 based data format that is openly documented, couldn't we try to convince them to contribute directly to the I/O plugin and maintain it?

[ericpre]

I have a version of GMS that can save dm5 on my laptop, so I can easily make file when necessary.

[ericpre]

https://www.gatan.com/dm5-documentation

[CSSFrancis]

If Gatan is already making the move to an hdf5 based data format that is openly documented, couldn't we try to convince them to contribute directly to the I/O plugin and maintain it?

If you would like to reach out to them that would be great!

[CSSFrancis]

pre-commit.ci autofix

[CSSFrancis]

I think this should be mostly good to go although there is some behavior (mainly multiple 1D Signals) That I don't have a good grasp on how that should be saved/represented.

There are lots of different pathways for reading data so I'm not entirely certain that I have handled them all correctly. As far as ROI's go I'm not sure if those should be added by default. Someone with more DM knowledge than me (I literally used Digital Micrograph for the first time 2 weeks ago) would be helpful 😅.

Also let me know if there is interest in a DM3/DM4 writer. I should be able to reverse engineer those now that I understand the structure a little bit better.

[jlaehne]

As far as ROI's go I'm not sure if those should be added by default

Well there have been discussions #22 and attempts concerning #1491 marker import for dm3/4 ... would still be a really nice feature also for the older file versions, but maybe optional and not as default.

[CSSFrancis]

@jlaehne I might come back to this once I make some changes upstream in hyperspy. Namely replacing the Navigator with a ROI and allowing multiple navigators. That should make things more similar to DM and make the 1:1 mapping a little easier.

[CSSFrancis]

@jlaehne The ROI's aren't terribly difficult to create although they appear to be ??unsupported?? with the free DM version?

[jlaehne]

I currently don't have access to the free version of DM ... but @aidanc151 has looked into that a bit and has a local implementation for reading these ROIs - so maybe you could team up - also he could check on the free DM version.

[ericpre]

I tried to make some dm5 file using GMS 3.6.0 and I had an issue with saving cropped data... so what I tried to save as dm4, then open in hyperspy and then saved in dm5 using hyperspy, which does work. These are the data:

When opening the data in GMS, it seems that the shape of the data is incorrect, hyperspy reads it correctly though! Some of the original metadata are lost, but I guess this is secondary at the moment.
Below are some cropped data:
EELS_SI.zip

The dm5 are made using:

import hyperspy.api as hs

fname = "EELS_SI"

s = hs.load(f"{fname}.dm4")
s.save(f"{fname}.dm5")
s.T.save(f"{fname}_transposed.dm5")

When reading the transposed file in GMS, it seems that there is an encoding issue with the µ:

[ericpre]

Also let me know if there is interest in a DM3/DM4 writer. I should be able to reverse engineer those now that I understand the structure a little bit better.

From hyperspy/ImportRPL#2 (comment) and having a quick look, https://github.com/mfm24/DM3Utils may be a good reference to implement writing dm4 file.

[CSSFrancis]

I tried to make some dm5 file using GMS 3.6.0 and I had an issue with saving cropped data... so what I tried to save as dm4, then open in hyperspy and then saved in dm5 using hyperspy, which does work. These are the data:

When opening the data in GMS, it seems that the shape of the data is incorrect, hyperspy reads it correctly though! Some of the original metadata are lost, but I guess this is secondary at the moment. Below are some cropped data: EELS_SI.zip

This is a good sign right? :) Saving the metadata is bit more of a challenge considering that both Hyperspy and DM both have pretty much arbitrary metadata so mapping 1:1 is pretty much impossible.

I just dump the hyperspy metadata into a branch and vice-versa although that isn't the best for the case of going from DM --> Hyperspy --> DM :)