Define a set of standard questions to ask of ML

[joeyklee]

It would be great/helpful to define a set of critical questions for students to define when approaching ML models and data:

e.g.

• "Accurate for whom?"
• "Accurate for which purpose?"
• "Appropriate in which context?"
• "(Un)Representative of what or whom?"

As a learning outcome, it would be great if students start to always ask or address these questions in their work, reflections, and future projects.

[shiffman]

This is great! It would be nice to brainstorm questions and discuss in class too. Would this make sense as a wiki page, additional README file in the repo, or. . .?

[shiffman]

Seems like tomorrow is the time to discuss this a bit? Would it make sense to fold some of these points into @lydiajessup's Data Wrangling write-up in #38.

[lydiajessup]

I think that would be great! Let's discuss tomorrow morning before class the best way to add them in. In particular, I think that some of these questions could be a great way to discuss the historical context of the way gender is classified in the Titanic dataset.

[shiffman]

I hesitate to close this as it's such an excellent reference and I didn't get every bit of language from the original suggestion. @joeyklee if you review the materials and feel it meets the spirit of what you suggested I'll let you click the button below. We can also leave it open as a reminder to ask these questions for the rest of the semester and in future iterations of the course.

[ellennickles]

For the archive! Datasheets for Datasets suggests questions to ask from these categories: motivation, composition, collection process, preprocessing/cleaning/labeling, uses, distribution, and maintenance

[lydiajessup]

This isn't a set of questions exactly, but ITP student Emma Norton (@doodybrains on github) just published this article that I think does an amazing job of explaining why these kinds of questions are important in the larger context of computing: https://reallifemag.com/bad-metaphors-close-to-the-metal/ I wasn't sure exactly where to put this but wanted to be sure to share!