Dataset inspiration

[davanstrien]

I thought it could be useful to collect some good places where you might potentially find interesting data to add as part of this sprint. I'll keep updating this list throughout the hackathon.

National library datasets

Many national libraries have pages that list data available for bulk access. Some examples:

• https://data.bnl.lu/data/
• https://data.bl.uk/
• https://data.nls.uk/
• https://www.bnf.fr/fr/bnf-datalab
• https://labs.loc.gov/lc-for-robots/
• http://data.cervantesvirtual.com/
• ...

[cceyda]

https://iiif.io/ API is used by some places but it is very documentation heavy... I think I saw a simpler python wrapper around it recently but can't find now.

[davanstrien]

iiif.io API is used by some places but it is very documentation heavy... I think I saw a simpler python wrapper around it recently but can't find now.

Was it maybe https://github.com/Princeton-CDH/piffle?

I agree IIIF can be great for ML-related things. I am currently working on a dataset (https://zenodo.org/record/6786970) that uses the label field from IIIF manifests as a noisy label. Not all IIIF images will have these but there are some collections where it might be possible to get quite a lot of annotated (if noisy) labels 'for free'.

[aih]

This is an excellent thread. Would it make sense to create a curated list in markdown in the repository itself, in the form of the "awesome" code series, e.g. https://github.com/vinta/awesome-python?

[davanstrien]

That sounds great! Did you have in mind a list of high-level sources for finding datasets?

It could also be worth checking if this existing awesome list was open to a section on data sources?

[aih]

I actually was thinking of a list of links to the datasets or potential datasets themselves, but the idea is to commit the list in Markdown to this repo, so it can grow and be version controlled. Once there's an Awesome list here (and a contribution guideline), people will make PRs.

[davanstrien]

My main concern with something like this is keeping it maintained and also making sure it remains useful as it grows i.e. has filters for metadata etc. One option we've discussed is trying to encourage the Hugging Face hub maintainers to make it easier to indicate the 'domain' a dataset is from so it would be possible to filter by 'lam' datasets + text classification. This would also allow tags to be added to datasets stored under different organizations. I'm planning to write up an issue about that tomorrow (I'll make sure to CC you). WDYT?

[aih]

Yes, a domain indicator would not just work for Library data sets, but also for others, and allow users to sort, filter and even classify :-). There is already a potential building, between repos on Github and datasets on Huggingface, to combine the two in a meaningful searchable way.

[davanstrien]

This resource from @mialondon is a great source of potential datasets

[cceyda]

Today's trending tweet and the related article
was using Science Museum Group. a nice collection of Creative Commons(specifically CC BY-NC-ND 4.0, CC BY-NC-SA 4.0 etc) licensed images & metadata

[davanstrien]

That would be interesting collection to work with! I have never used it but the Science Musuem does have an API (https://www.sciencemuseumgroup.org.uk/about-us/collection/using-our-collection-api/) which could be used to either do a one type search to collect material or could potentially be called inside a loading script based on some user-specified parameters.

The Science Museum has recently worked on some exciting ML projects: https://www.sciencemuseumgroup.org.uk/project/heritage-connector/.