Support linking to datasets in external repositories?

[shiltemann]

I am creating an ARC for a project with very large data (>750GB raw sequencing data). This data is all already uploaded to SRA, and it seems wasteful to include it all again in the ARC and take up storage space unnecessarily.

What is the ARC philosophy is here, should we always include all the raw data directly in the ARC, or do we want to support linking to datasets from some of these 'trusted' public repositories as well (e.g. in assay metadata in place of the file path)? Many journals require authors to submit raw data to one of these repositories (SRA, ENA, etc) so it may be a common use case in the community.

[Freymaurer]

Hello! This discussion might be more fitting on the ARC specification repo. I will transfer it 🙂

[Freymaurer]

@HLWeil @muehlhaus

[HLWeil]

Hey, @shiltemann
we definitely plan to include the possiblity to link to external data sources.

I guess the most basic constraint there would be, that the data has to be publicly accessible. Then the Input/Output [Data] column could be used as is and instead of local paths, the direct URIs to the raw data content could be linked.

[shiltemann]

@HLWeil that's great, indeed it should be publicly accessible, and maybe some confidence that it will stay this way? e.g. a doi link or by defining some trusted repositories?

On a practical note, do you have a feeling for the timeline of this feature? E.g. we hope to publish this paper before the end of the year, and would love to cite our ARC in the paper, do you think it might be supported by then or?

[shiltemann]

oh! do you mean I can already put a URL in the Output[Data] column of my assay? Last time I tried the CI tests failed because it wasn't a local file, but maybe I should have tried more recently. Or do you mean something different?

[Freymaurer]

The CI tests are another thing! I am not sure about this. Sadly @kMutagene is currently on vacation. Maybe @HLWeil knows more.

[HLWeil]

Yup, was kinda hoping this would be the case, as it means the tests are working 😄

I think adding the URL data to the specification and updating the tests accordingly is something we could realize in the upcoming months. I could start with the specs part before @kMutagene is back from vacation to update the tests.

@shiltemann, IMO you're safe to just include the URLs and ignore the tests for now, if you don't mind the red X.

[shiltemann]

Awesome 🎉 Ok then I will populate my ARC with the URLs and keep an eye on the CI updates 👍

thanks both!

[kMutagene]

@shiltemann could you link a failed validation pipeline?