[COMP599_Fall2021] Design and Build Intelligent Systems

General Information

Instructor	Jin Guo
TA	Deeksha Arya
Class Time	TR 2:35 am-3:55 pm
TA Office Hours	Zoom (Tuesday 9:20-10:20am)
Location	BURN 1B24
Discussion Forum	Slack

• I normally design many in-class activities for upper-level classes to motivate discussion and collaborative learning. However, given the risks of going to a classroom and other constraints you might have, I will allow people to join remotely through Zoom for the first few weeks. The zoom session will also be recorded. We will adjust as the semester goes.

• Please fill in this background survey and topic preference as discussant before Sept 7th.

Description

This course is going to explore how to design and build an intelligent system from a software engineering perspective, from requirement gathering and analysis to deployment and maintenance. We will also touch AI ethics and its implications to design.

Prerequisite

While there are no official prerequisite courses, you will enjoy and appreciate this course more if you have taken COMP303, COMP424 and COMP551 already.

Reference Material

We will not concentrate on any particular resources. Instead, the readings will include content from book chapters, research papers, blog posts, talks, etc. The pointers to those content will be added to the schedule later.

• Books:
  • Thinking in Systems: A Primer (Chapter scans will be shared through OneDrive links)
  • Human Compatible: AI and the Problem of Control (Chapter scans will be shared through OneDrive links)
  • Design Justice (open access)
  • Building Intelligent Systems A Guide to Machine Learning Engineering (access through McGill Library)

Assessment and Evaluation (Tentative)

Assessment Method	Weight
Participation (inclass and online)	10%
Assignment	60%
Final Project	30%

• Any form of plagiarism, cheating is strictly banned during midterm or final exam. Integrity is crucial to this course and your future career. Any violation against academic integrity will be taken very seriously. For more information, please refer here.

Schedule (Tentative)

Subject to adjustments

Lecture	Date	Content	Reading	Note	Discussant
1	2 Sep	Introduction	BIS book: Chapter 1, 2 TIS book: Intro (Onedrive)
2	7 Sep	Intro to Modern Software Engineering	A collection of videos by Sommerville Ian about fundatoinal SE Quality Attributes
3	9 Sep	Intelligence and ML	BIS book: Chapter 16, 17, 18 Human Compatible: Intelligence (Onedrive)	Assignment-1 (Due 20 Sep)
4	14 Sep	Model Quality	BIS book: Chapter 19, 20 How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation Beyond Accuracy: Behavioral Testing of NLP Models with CheckList Model Cards for Model Reporting
5	16 Sep	From Model to System	Software Engineering for Machine Learning: A Case Study Hidden Technical Debt in Machine Learning Systems TIS book: Chapter 4 Why Systems Suprise Us (Onedrive)		Saskia
6	21 Sep	Data Acquisition & Management	BIS book: Chapter 9 A Survey on Data Collection for Machine Learning: A Big Data - AI Integration Perspective		Luca
7	23 Sep	Requirement and AI - 1	Requirements Engineering for Machine Learning: Perspectives from Data Scientists		Ada
8	28 Sep	Requirement and AI - 2	Keynote talk by Amy Ko at RE 2021
9	30 Sep	Team and Collaboration	How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study Data Scientists in Software Teams:State of the Art and Challenges		Martin
10	5 Oct	Data Quality	The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction BIS book: Chapter 15
11	7 Oct	System Quality		Assignment-2 (Due 17 Oct)	Avinash
12	15 Oct (READING WEEK MakeUp Class)	Continous Delivery Design for Human-AI Interaction (UX)	Guidelines for Human-AI Interaction Human-Centered Artificial Intelligence: Three Fresh Ideas	Project Milestone-1 (Due 26 Oct)
13	19 Oct	Design - Visualization	Will you accept an imperfect AI? exploring designs for adjusting end-user expectations of AI systems When (ish) is my bus? user-centered visualizations of uncertainty in everyday, mobile predictive systems The state of the art in integrating machine learning into visual analytics
14	21 Oct	Design - Decision-making	The Principles and Limits of Algorithm-in-the-Loop Decision Making Judgment under uncertainty: Heuristics and biases
15	26 Oct	Design - Learning	ArgueTutor: An Adaptive Dialog-Based Learning System for Argumentation Skills Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols
16	28 Oct	Design - Creativity	Creativity support tools: accelerating discovery and innovation AI as Social Glue: Uncovering the Roles of Deep Generative AI during Social Music Composition	Assignment-3 (Due 5 Nov)
17	2 Nov	Design - Inclusion	Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors Design Values: Hard-Coding Liberation?
18	4 Nov	Design - Inclusion (cont'd) AI principles Overview	Automated generation of storytelling vocabulary from photographs for use in AAC Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-based Approaches to Principles for AI
19	9 Nov	Safety	Autonomous Vehicle Safety: An Interdisciplinary Challenge An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software	Project Milestone-2 (Due 18 Nov)	Roman
20	11 Nov	Security	The AI-Based Cyber Threat Landscape: A Survey
21	16 Nov	Privacy	SoK: Towards the Science of Security and Privacy in Machine Learning Designing privacy-aware internet of things applications		Percy
22	18 Nov	Accountability/Auditing	Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims (Section 2 and 3)		Hashem Félix
23	23 Nov	Transparent and Explainability	Explainable machine learning in deployment Designing Theory-Driven User-Centric Explainable AI		Souleima Oumar
24	25 Nov	Fairness	Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI		Kevin
25	30 Nov	Fairness and Wrap up	“The Human Body is a Black Box”: Supporting Clinical Decision-Making with Deep Learning
26	10 Dec (Zoom)	Presentation

Credit:

The content regarding engineering aspects is greatly inspired by CMU 17-445/645: Software Engineering for AI-Enabled Systems which is developed by Christian Kästner et. al.

License

Unless otherwise noted, the content of this repository is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.