FAQs

[davanstrien]

Are there things not covered in the FAQs that you want to know? We'll use this thread to answer other questions and migrate them to the wiki if they seem relevant to a wider audience.

[mialondon]

Channelling some questions I can imagine GLAM folk asking:

• What's HuggingFace? (different levels of explanation might be useful here - the one sentence, jargon-free summary, the 'copy and paste this for your tech people' summary, ditto for marketing/collections management/board people
• Who's behind the hackathon?
• What kinds of things might people do with our data?
• What kinds of data would people produce, and would any of it be useful for us?
• Is there any information we can provide to make our data more useful/usable?
• Can we get any stats on uses of our data?
• Will people credit us?
• If we don't want our data used, what do we do?
• If we want to feature uses of our data, how do we get in contact with the right people?

[mialondon]

Another:

• Our data is licensed for NC (non-commercial) use. Is that ok?
• Our data is licensed for BY (attribution) use. Is that ok?

[mialondon]

• How long will HuggingFace keep our data?
• How is HuggingFace funded?

[davanstrien]

Thanks for these! I'll work on answering those in the FAQ over the next couple of days : )

[davanstrien]

Suggested response for one of these. Would be happy to get feedback before I add to the FAQs.

Is there any information we can provide to make our data more useful/usable?

Yes! Some of this information will be easy to provide whilst others will be more challenging. Some fundamental things that can are important to document:

• The license under which the data is shared. Whilst making a dataset 'open' by making it publicly accessible/downloadable on the web is important, many people will not want to work without a license telling people what they can and can't do with the data.
• Use a standard license: unless you have a very good reason to create a bespoke licenses, it is better to use an existing licence. Creative Commons offer a range of licenses which allow you to share datasets under different conditions. Creative Commons licenses have been tested in court.

Beyond this, there are many things you could consider documenting about your data. When we add data to the Hub, we aim to complete Dataset Cards for that dataset. These Dataset cards aim to provide potential data users with a better understanding of its limitations and biases. Datasheets for Datasets offers some more background on data cards.

One way to approach your documentation is to read through the guidance Dataset Card Creation Guide and think about whether someone working with your dataset could find out this information.

It is also helpful to think about aspects of the data and the collections from which they were drawn, which may be known to (some) staff working in your institution but may not be evident from looking at the data. For example, many institutions will exclude some books from mass digitization efforts. This could be because of copyright, size, damage to the item, or for some other reason(s). These factors may not be evident to an external user of this dataset. Trying to document this internal knowledge could be very helpful for people to understand how to approach your dataset.

One final important consideration is if your data has particular biases that people should be aware of. The fact that data has bias doesn't necessarily mean it shouldn't be shared, but it is essential to convey this to potential users. especially as machine learning models trained on this data has the potential to proliferate these biases.

[mialondon]

That's looking good to me, thanks @davanstrien! I love the license information.

What about tweaking this sentence (partly to avoid repeating 'aims'): 'These Dataset cards aim to provide potential data users with a better understanding of its limitations and biases.' to 'Dataset cards help potential data users understand the limitations and biases in your data.'

Also change 'books' to 'items'?

And maybe add something about 'bias' so that readers have a sense of how it's used in this context. Perhaps something like this after the first 'One final important' sentence?:

'People who work with data often talk about data 'bias'. In this context, 'bias' refers to the various ways in which data can reflect decisions about what was included or excluded, or the context in which items were described or labelled.'

[mialondon]

And a comment on an existing answer, which I'd read previously but understood differently after hearing from @davanstrien:

I work for a LAM institution: we already have data hosted elsewhere. Can we still expose our data via the Hugging Face Hub?

Yes, it's possible to use a 'dataset script' to make your dataset available. This dataset script can point to the files hosted in your repository. See guidance on creating a dataset loading script for more information on doing this.

I'd read 'point to' as 'redirecting to files in their current location' but maybe it's actually 'point a dataset script to files hosted in your repository, so that they can be copied over to HF'.

Could 'make your dataset available' be more precise about the process? e.g. use a 'dataset script' to copy your dataset over to HuggingFace in an appropriate format.

[Skorkmaz88]

@mialondon I think one example from Hugginface is : a computer vision dataset. In this example, images are not actually hosted in huggingface but image names and associated data saved in CSV files, such that training process would need to download them on fly.

[Skorkmaz88]

Yet before I forget, this comes with one potential downside, if the resource location changes, with no indication, then the dataset script will fail in the future, from what I had seen in the archives of Smithsonian and Getty, their resources stayed in the same routes for a long time so this was not an issue when I was fetching their data.

[davanstrien]

As @Skorkmaz88 for most dataset script, the files don't get copied to the hub. The interaction with the hub comes when you call load_dataset.

You could use load_dataset with a local script something like,

ds = load_dataset('mydata.py')

where mydata.py is a Python file in your local directory which contains a dataset script.

If you instead pass load_dataset the id for a dataset script hosted on the hub the main difference is that the datasets library will get the dataset script from the repository.

For example if you load this dataset: https://huggingface.co/datasets/biglam/atypical_animacy with

dataset = load_dataset("biglam/atypical_animacy")

The datasets library will grab this file (https://huggingface.co/datasets/biglam/atypical_animacy/blob/main/atypical_animacy.py) from the hub and then download the actual data onto your machine (or wherever you are running the code) from the British Library repository. This URL is specified in the loading script here: https://huggingface.co/datasets/biglam/atypical_animacy/blob/main/atypical_animacy.py#L56