fall-2018

This is the archived site for the Fall 2018 offering of this course. Go to the current offering here.

Overview

Empirical methods play a key role in the evaluation of tools and technologies, and in testing the social and technical theories they embody. No matter what your research area is, chances are you will be conducing empirical studies as part of your research. Are you looking to evaluate a new algorithm? New tool? Analyze (big) data? Understand what challenges practitioners face in some domain? This course is a survey of empirical methods, appropriate for all computer science PhD students, including Software Engineering and Societal Computing.

This course provides an overview and hands on experience with a core of qualitative and quantitative empirical research methods, including interviews, qualitative coding, survey design, and large-scale mining and analysis of data. Students will mine and integrate data from and across online software repositories (e.g., GitHub and Stack Overflow) and employ a spectrum of data analysis techniques, ranging from statistical modeling to social network analysis.

There will be extensive reading with occasional student presentations about the reading in class, weekly homework assignments, and a semester-long research project for which students must prepare in-class kickoff and final presentations as well as a final report.

After completing this course, you will:

• become a more sophisticated consumer of empirical research, both in your field and outside
• develop the methodological skills that can help you design and carry out empirical components in your own research program
• be able to analyze empirical data, draw conclusions, and present results
• be able to read, summarize, and present academic papers

Coordinates

Th 09:00 - 11:20 a.m. in Wean 5328

Course materials and assignments on Canvas

Bogdan Vasilescu
[email protected]
WEH 5115

Course Syllabus and Policies

The syllabus covers course overview and objectives, evaluation, time management, late work policy, and collaboration policy.

Learning Goals

The learning goals describe what I want students to know or be able to do by the end of the semester. I evaluate whether learning goals have been achieved through assignments, written project reports, and in-class presentations.

Schedule

We cover the following topics (slides or notes posted when available):

Date	Topic	Deadlines
08/30	Introduction
09/06	Literature Review and Theory	HW1 due (comparison of methods)
09/13	Interviews	HW2 due (literature review)
09/20	Grounded Theory	HW3 due (interviews)
09/27	Surveys	HW4 due (grounded theory)
10/04	Introduction to Measurement (no slides)	HW5 due (survey)
10/11	Your Research Project Proposal (no slides)
10/18	Experimentation	HW6 due (hypothesis testing)
10/25	Quasi-experimentation	HW7 due (experiment)
11/01	Time Series Analysis	HW8 due (regression)
11/08	Mixed-methods (no slides)	HW9 due (time series analysis)
11/15	Text Mining (slides by David Blei)	HW10 due (mixed methods)
11/22	No Class - Thanksgiving
11/29	Social Network Analysis	HW11 due (text mining)
12/06	Final Presentations (no slides)	HW12 due (social network analysis)
12/13		Final project report due

Readings

Introduction (8/30)

We will start out by looking broadly over the range of empirical methods you might consider using, and the assumptions and philosophical points of view they rely on. We also want to hear about the kinds of research problems you are working on or plan to work on, and the sorts of empirical questions they give rise to. Please be prepared to share this with the class, as it will help us to have fruitful discussions if we know a little about each others' research areas. We will also use the information to customize the course a bit to emphasize things for which there is a clear need.

Methods:

🔹 Chapter 1 from Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. Sage publications. (philosophical world views)

Literature Review and Theory (09/06)

Methods:

🔹 Stol, K.-J., & Fitzgerald, B. (2015). Theory-oriented software engineering. Science of computer programming, 101, 79-98.

🔹 In the following paper, read up to "DESIGNING RECOMMENDERS FOR TWITTER" on p. 1187:

Chen, J., Nairn, R., Nelson, L., Bernstein, M., & Chi, E. (2010). Short and tweet: experiments on recommending content from information streams. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.

🔹 In the following paper, read up to METHODOLOGY AND DATA SOURCES on page 302:

Mockus, A., Fielding, R. T., & Herbsleb, J. D. (2002). Two case studies of open source software development: Apache and Mozilla. ACM Transactions on Software Engineering and Methodology, 11(3), 309-346.

Discussion points:

Be prepared to contrast these two literature reviews. In each case, how much prior work was published? What kinds of gaps and questions were the papers addressing? How did the authors choose the papers they discussed, and the specific points they focused on?

Interviewing (9/13)

Methods:

🔹 Paul Goodman's (2005) Building Effective Interviewing Skills.

🔹 King, N. (2004). Using interviews in qualitative research. In C. Cassell & G. Symon (Eds.), Essential Guide to Qualitative Methods in Organizational Research (pp. 11-22). Loondon: Sage.

🔹 Seidman, I. (2012). Interviewing as qualitative research: A guide for researchers in education and the social sciences: Teachers college press. (Ch 4).

🔹 Seidman, I. (2012). Interviewing as qualitative research: A guide for researchers in education and the social sciences: Teachers college press. (Ch. 6).

Examples:

• Grinter, R. E., & Palen, L. (2002). Instant messaging in teen life, Paper presented at the 2002 ACM Conference on Computer-Supported Cooperative Work (pp. 21-30): ACM.

• Dabbish, L., Stuart, C., Tsay, J., & Herbsleb, J. (2012). Social Coding in GitHub: Transparency and Collaboration in an Open Software Repository. Paper presented at the 2012 ACM Conference on Computer-Supported Cooperative Work, Seattle, WA.

• Manotas, I., Bird, C., Zhang, R., Shepherd, D., Jaspan, C., Sadowski, C., . . . Clause, J. (2016). An empirical study of practitioners' perspectives on green software engineering. Paper presented at the International Conference on Software Engineering (ICSE), Austin, TX.

Grounded Theory (9/20)

Methods:

🔹 Miles, M.B, Huberman, A.M., & Saldana, J. (2014) Qualitative Data Analysis: A Methods Sourcebook. 3d Ed. Sage: Los Angeles.:

• Ch. 4: Fundamentals of Qualitative data analysis.
• Ch. 11: Drawing and Verifying Conclusions.

Examples:

• Razavi, M. N., & Iverson, L. (2006). A grounded theory of information sharing behavior in a personal learning space, Proceedings of the ACM Conference on Computer Supported Cooperative Work (pp. 459-468).

• de Souza, C. R., & Redmiles, D. F. (2008). An empirical study of software developers' management of dependencies and changes, Proceedings of the 30th International Conference on Software Engineering (pp. 241-250).

• Deterding, S. (2016). Contextual autonomy support in video game play: a grounded theory. Paper presented at the Conference on Human Factors in Computing Systems (CHI).

Surveys (9/27)

Methods:

🔹 Chapters from Dillman, D., Smyth, J. D., & Christian, L. M. (2014). Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method (4th ed.). Hoboken, NJ: Wiley.

• Chapter 1: Sample Surveys in our Electronic World
• Chapter 2: Reducing People's Reluctance to Respond to Surveys
• Chapter 4: The Fundamentals of Writing Questions
• Chapter 5: How to Write Open and Closed Ended Questions

Examples:

• Henne, B., Harbach, M., & Smith, M. (2013). Location privacy revisited: factors of privacy decisions. Extended Abstracts on Human Factors in Computing Systems.
• Shklovski, I., Mainwaring, S. D., Skúladóttir, H. H., & Borgthorsson, H. (2014). Leakiness and creepiness in app space: perceptions of privacy and mobile app use, Proceedings of the ACM conference on Human factors in computing systems (pp. 2347-2356): ACM.
• Subramanyam, R., Weisstein, F. L., & Krishnan, M. S. (2010). User participation in software development projects. Communications of the ACM, 53(3), 137-141.

Introduction to Measurement (10/4)

Methods:

🔹 Chapter 10 from C. Wohlin et al., Experimentation in Software Engineering, Springer-Verlag Berlin Heidelberg 2012

🔹 Chapter 6 from F. Shull et al. (eds.), Guide to Advanced Empirical Software Engineering. Springer 2008 (similar content as the Wohlin chapter but slightly different presentation; read one or the other)

🔹 Chapter 6 from MacKenzie. Human-Computer Interaction. Elsevier 2013

Optional readings:

🔹 Lawrence, N. W. (2007). The basics of Social Research. Qualitative and Quantitative Approaches:

• Chapter 5 - Measurement
• Chapter 10 - Analysis of data

Example papers for in class presentations:

• Filippova, A., Trainer, E., & Herbsleb, J. D. (2017). From diversity by numbers to diversity as process: supporting inclusiveness in software development teams with brainstorming. In Proceedings of the 39th International Conference on Software Engineering (pp. 152-163). IEEE. [focus on the quantitative analysis of survey responses]
• Vasilescu, B., Filkov, V., & Serebrenik, A. (2015). Perceptions of diversity on GitHub: A user survey. In Proceedings of the Eighth International Workshop on Cooperative and Human Aspects of Software Engineering (pp. 50-56). IEEE. [focus on the quantitative analysis of survey responses]
• Kaptein, M., & Robertson, J. (2012, May). Rethinking statistical analysis methods for CHI. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 1105-1114). ACM. [focus on the threats to validity and ways to mitigate]

Experiments (10/18)

Please read the SCC chapters in preparation for the lecture. There are no paper presentations assigned, we will discuss the examples in class.

Methods:

🔹 Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference: Wadsworth Cengage learning:

• Chapter 1
• Chapter 2
• Chapter 8

Example papers:

• Tomkins, A., Zhang, M., & Heavlin, W. D. (2017). Single versus double blind reviewing at WSDM 2017. arXiv preprint arXiv:1702.00502.

Plus the following, in this order:

• Sobel, A. E. K., & Clarkson, M. R. (2002). Formal methods application: An empirical tale of software development. IEEE Transactions on Software Engineering, 28(3), 308-320.

• Berry, D. M., & Tichy, W. F. (2003). Comments on "Formal methods application: an empirical tale of software development". IEEE Transactions on Software Engineering, 29(6), 567-571.

• Sobel, A. E. K., & Clarkson, M. R. (2003). Response to "Comments on 'Formal methods application: an empirical tale of software development'". IEEE Transactions on Software Engineering, 29(6), 572-575.

Quasi-experimental Design & Linear Regression (10/25)

Methods:

🔹 Woolridge, J. M. (2003). Introductory econometrics: A modern approach. Thomson, Mason. Chapter 2 - Simple Regression [skim]

🔹 F.E. Harrell, Jr., Regression Modeling Strategies, Springer Series in Statistics, Chapters 1&2 - Regression general aspects: [Chapter 1: skim] [Chapter 2: read 2.1--2.3, 2.7]

Optional reading:

🔹 Shadish, Cook, & Campbell, Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Chapter 3, Construct Validity and External Validity.

🔹 Oktay, H., Taylor, B. J., & Jensen, D. D. (2010, July). Causal discovery in social media using quasi-experimental designs. In Proceedings of the First Workshop on Social Media Analytics (pp. 1-9). ACM.

Examples:

• Sinatra, R., Wang, D., Deville, P., Song, C., & Barabási, A. L. (2016). Quantifying the evolution of individual scientific impact. Science, 354(6312), aaf5239.
• Lim, S. (2009). How and why do college students use Wikipedia? Journal of the Association for Information Science and Technology, 60(11), 2189-2202.
• Bird, C., Nagappan, N., Devanbu, P., Gall, H., & Murphy, B. (2009). Does distributed development affect software quality? An empirical case study of Windows Vista. Communications of the ACM, 52(8), 85-93.

Time Series Analysis (11/1)

Methods:

🔹 Cowpertwait, P. S., & Metcalfe, A. V. (2009). Introductory time series with R. Springer Science & Business Media. [great practical book with applications in R; read selectively for topics you're interested in and otherwise keep for reference; decompositions, e.g., seasonality+trend, are particularly useful]

🔹 Woolridge, J. M. (2003). Introductory econometrics: A modern approach. Thomson, Mason. Chapter 10 - Time series [read if you want to see how the sausage is made, otherwise keep for reference]

🔹 Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference: Wadsworth Cengage learning:

• Chapter 6 - Interrupted time series + Chapter 7 - Regression discontinuity design [ITS/RDD is a really powerful technique and the main topic of this lecture; skim the chapter to understand the main idea, the figures are particularly informative; I'll also give a short presentation in class]

🔹 Wagner, A. K., Soumerai, S. B., Zhang, F., & Ross‐Degnan, D. (2002). Segmented regression analysis of interrupted time series studies in medication use research. Journal of clinical pharmacy and therapeutics, 27(4), 299-309. [great example of how to apply the technique; read carefully after you've skimmed Shadish]

Examples to discuss in class:

• Kenmei, B., Antoniol, G., & Di Penta, M. (2008). Trend analysis and issue prediction in large-scale open source systems. In Software Maintenance and Reengineering, 2008. CSMR 2008. 12th European Conference on (pp. 73-82). IEEE.

• Trockman. A., Zhou, S., Kästner, C., & Vasilescu. B. (2017). Adding Sparkle to Social Coding: An Empirical Study of Repository Badges in the npm Ecosystem. [there's a lot in this paper, focus only on one example of applying ITS/RDD, I recommend dependency management]

• Jadidi, M., Karimi, F., & Wagner, C. (2017). Gender Disparities in Science? Dropout, Productivity, Collaborations and Success of Male and Female Computer Scientists. arXiv preprint arXiv:1704.05801.

Mixed-Method Design (11/8)

Methods:

🔹 Creswell Chapter 10

🔹 Venkatesh, V., Brown, S. A., & Bala, H. (2013). Bridging the qualitative-quantitative divide: Guidelines for conducting mixed methods research in information systems. MIS quarterly, 37(1), 21-54.

🔹 Onwuegbuzie, A. J., & Collins, K. M. (2007). A typology of mixed methods sampling designs in social science research. The qualitative report, 12(2), 281-316. [skim only; good discussion of how to select sample sizes for mixed-methods research, depending on the study goals]

Examples: In your presentations in class, describe clearly: which methods are being mixed in the paper; how are the different methods combined; which threats to validity of each method does the mixture alleviate.

• Greiler, M., Deursen, A. V., & Storey, M. A. (2012). Test confessions: a study of testing practices for plug-in systems. In Proceedings of the 34th International Conference on Software Engineering (pp. 244-254). IEEE Press.
• Mamykina, L., Manoim, B., Mittal, M., Hripcsak, G., & Hartmann, B. (2011). Design lessons from the fastest Q&A site in the West. In Proceedings of the SIGCHI Conference on Human factors in Computing Systems (pp. 2857-2866). ACM.
• Trockman, A., Zhou, S., Kästner, C., & Vasilescu, B. (2018). Adding Sparkle to Social Coding: An Empirical Study of Repository Badges in the npm Ecosystem. In Proceedings of the 40th International Conference on Software Engineering (pp. 511-522). [there's a lot in this paper, focus on how the qualitative and quantitative methods were mixed]

Text Mining (11/15)

Methods:

🔹 Manning, C. D., & Schütze, H. (1999). Foundations of statistical natural language processing (Vol. 999). Cambridge: MIT press. - Chapter 1 [skim, interesting background reading]

🔹 Bird, C., Menzies, T., & Zimmermann, T. (Eds.). (2015). The Art and Science of Analyzing Software Data. Elsevier:

• Chapter 3 - Analyzing text in software projects [skim Section 3.4, which gives an overview of different automatic text mining methods]
• Chapter 6 - Latent Dirichlet Allocation [practically oriented coverage of LDA; read carefully]
• Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of language and social psychology, 29(1), 24-54. [skim, LIWC is a very influential piece of work]

Examples to discuss in class:

• Asuncion, H. U., Asuncion, A. U., & Taylor, R. N. (2010, May). Software traceability with topic modeling. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1 (pp. 95-104). ACM.
• Wang, Y. C., Kraut, R., & Levine, J. M. (2012, February). To stay or leave?: the relationship of emotional and informational support to commitment in online health support groups. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (pp. 833-842). ACM.
• Tan, C., & Lee, L. (2015, May). All who wander: On the prevalence and characteristics of multi-community engagement. In Proceedings of the 24th International Conference on World Wide Web (pp. 1056-1066).

Social Network Analysis (11/29)

Methods:

🔹 From "Network Science" by Albert-László Barabási. Cambridge University Press, 2016:

• Chapter 1 - Introduction [light read, talks about why networks are important]
• Chapter 2 - Graph Theory [light read, introduces terminology]
• Chapter 9 - Communities [skim, community detection has tons of applications in our research]

🔹 From "Networks, Crowds, and Markets: Reasoning about a Highly Connected World." by David Easley and Jon Kleinberg. Cambridge University Press, 2010:

• Chapter 2 - Graphs [some overlap with Barabási's Chapter 2, skim]
• Chapter 3 - Strong and Weak Ties [important, skim/read carefully]
• Chapter 4 - Homophily [important, skim/read carefully]

Examples to discuss in class:

• Bird, C., Pattison, D., D'Souza, R., Filkov, V., & Devanbu, P. (2008). Latent social structure in open source projects. In Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering (pp. 24-35). ACM.
• Backstrom, L., & Kleinberg, J. (2014). Romantic partnerships and the dispersion of social ties: a network analysis of relationship status on Facebook. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 831-841). ACM.
• Way, S. F., Larremore, D. B., & Clauset, A. (2016). Gender, Productivity, and Prestige in Computer Science Faculty Hiring Networks. In Proceedings of the 25th International Conference on World Wide Web (pp. 1169-1179). International World Wide Web Conferences Steering Committee.

Additional examples:

• Clauset, A., Arbesman, S., & Larremore, D. B. (2015). Systematic inequality and hierarchy in faculty hiring networks. Science Advances, 1(1), e1400005.