Choose a format for publishing harmonized datasets

[gaurav]

We currently have an initial, incomplete set of harmonized data from some IDC example datasets (#4), and need to chose a format to store and publish this data in a way that maintains provenance information. Here are some possible formats:

• CEDAR instances
• Portable Format for Biomedical Data (PFB) (see also the description on Bento-Meta)

Please feel free to add any other formats we should look at!

This issue covers comparing these formats for our purposes and to translate the incomplete harmonized data into that format as a potential exemplar.

[fedorov]

Before we answer this question - do you know where will the resulting files live within the CRDC? What resource is going to be responsible for supporting users in querying the data stored in those files?

[balhoff]

I think the answer to that lies in the ultimate architecture for CDA. Definitely the querying part; not sure about storage of the files.

[fedorov]

Since CDA was mentioned - adding @DavidPotCanuck who is one of the leads for CDA, in case he is interested to participate in this discussion.

[gaurav]

I've made a preliminary conversion from the ISPY1 patient clinical subset to PFB by using a JSON file that records the mappings. You can download the PFB file, or have a look at their internal representation extracted with the PyPFB tool: the Avro schema, PFB metadata node, and data.

The good news is that PFB can be used to store textual, numerical and enumerated data, along with a reference to the caDSR CDE used for the mapping (from which we could map the harmonized values to NCIt concept identifiers). The bad news is that Avro fields and enum values are both restricted to the regex [A-Za-z_][A-Za-z0-9_]*. The PFB developers have come up with a simple representation for inserting Unicode characters (using e.g. _00A9_ to represent U+00A9 = ©), but PyPFB doesn't consistently translate those field names into and out of JSON at the moment. We could probably fix this without too much of a problem, assuming that this is a bug in PyPFB and not a misinterpretation on my part. PFB doesn't currently have a facility for storing verbatim values, so we do lose the actual values recorded in the original data. We might be able to propose a change to PFB to record this information if it is important to us. Another possibility is that we could store all the mapping information in another part of the Avro file, leave the verbatim values as-is and only convert them into harmonized values when exporting it.

I'm going to leave this conversion as-is for now and move on to trying to convert the same dataset into the CEDAR instance format so we can do a side-by-side comparison between these two formats.

[gaurav]

We'll have some example output files in cancerDHC/example-data#10 -- we can use that to figure out which format work best for our needs.

[gaurav]

For our immediate needs, the LinkML instance format (in YAML) seems to be a good representation, and has been developed into some exemplars by the CCDH Data Model Harmonization team as part of the CCDH Pilot. Future formats will probably be supported by adding generators to LinkML, so that data from any LinkML model can be converted into that format (e.g. see the issue tracking an Avro/PFB generator for LinkML). Given that, I think we can close this issue until specific use-cases emerge from CDA and the CRDC nodes.