JSON lines / BSON and other non-flat data formats

[ivbeg]

Hi! I am working on ETL and data platform with NoSQL as primary data formats. Most data is JSON lines, BSON and Parquet converted from XML, JSON and extracted from REST API.

I would like to add data package specification as one of ETL destinations. But I can't find how to add NoSQL schemas into data package. So our team re-created data packaing logic implemented for NoSQL. Here are examples of packages resprojects-resdata-2021-7-28-22-52.zip and datamos-7704782036-214FZ-2021-10-23-7-37.zip

The differences are:

• we use collections not tables to describe data (it's MongoDB logic)
• we have simpified schema to define data fields
• most data files inside out packages are compressed using
• we added documentation as part of data description

It was a temporary solution, I am not sure that we need to continue develop our way of data packaging in parallel. So maybe it's possible to reimplement similar logic using Data Package spec?

Is it possible to add support o JSON lines and BSON files? Is it possible to extend data package spec to support similar usage?