Multilanguage Support and Compatibility with Open Science

[andrew-saydjari]

Given the plans for this format to be widely used for the Roman, I was wondering what the plans were for multi-language support? FITS and HDF5 have wrappers for the low-level libraries in IDL, Python, Julia, R, Rust, etc. with the performance features (chunking, distributed/parallel read/write) needed to handle large scale community reprocessing. Without any plans to add such features and multi-language support, it seems like transitioning to ASDF would be premature and be in opposition to NASA TOPS.

[nden]

Thank you for your interest!
ASDF is a general format used by software packages and missions, one of which is Roman. Please send questions for Roman plans to the Roman help desk. The preferred way is to contact https://romanhelp.stsci.edu)https://romanhelp.stsci.edu . However, if you don't have a MyST account you can use [email protected].
The documentation associated with this Github organization has considerable discussion comparing formats and listing the benefits of ASDF. There's no contradiction with NASA TOPS, all development is open and accessible to everyone.
As for other languages, which one are you most interested in and why? How do you intend to use data in the ASDF format?

[andrew-saydjari]

Open science is not just about open development, but also about accessibility. One of my biggest concerns here is about the internal serialization, which is almost always a programming language (and often minor version) dependent operation. I am very concerned that by leaning on internal serialization of Python types, ASDF, is fundamentally exclusive to the large community of open source developers who do not work in Python.

My personal interest is most focused on Julia. It is my programming language of choice and I have written major pipelines for both large imaging and spectroscopic surveys in that language, happily interfacing with FITS and HDF5 and other portable file formats. I anticipate having to interact with ASDF in the development pipelines for processing of the Galactic Roman Plane Survey.

I have been through the documentation and the accompanying paper, but don't really see a clear table that compares ASDF to the other major file formats. Several of the functionalities (beyond language interoperability) I mentioned in the OP are present in those file formats but not in ASDF, which will seriously limit the performance of major external pipelines (which are essential for open science).