GU6008 Experimental Methods for Social Research

Fall 2021 | 2:10-4:00pm Fridays

Room: TBD

Office hours: 1-2:30pm Tuesdays, please sign up here

Instructor Email: jyc2163@columbia.edu

---

Course Summary

What is this course about?

An experiment is a data collection strategy that involves randomization and control. When properly designed, experiments enable unbiased estimation of causal effects in a sample. The goal of this course is to introduce you to the logic and practice of experiments. This course has two parts. The first part is an introduction to experiments and their assumptions. You will build upon this knowledge in the second part of the course to design an experiment on a topic of your choice.

Why learn experimental methods for social research?

If you're in graduate school, you are likely interested in identifying and answering important social scientific questions. Experimental methods can help you do that. Critical examination of exemplary experiments will uncover new questions that you might be interested in tackling, and you'll learn how to conduct your own experiment, identify its strengths and weaknesses, and analyze experimental results thoughtfully.

Even if you do not plan to use experiments in your own research, experiments are increasingly used in the social sciences. You will want to be able to read, understand, and critique experiments. You may also want to know the limitations of experimental research to better motivate other methodological choices.

Finally, experimental methods are a foundation for learning more advanced methods. We will not be covering quasi-experimental designs in this course, but a clear understanding of experimental methods is necessary to understand other models for causal inference.

What will you be able to know and do by the end of the course?

1. Identify the logic of causality assumed in experiments.
2. Evaluate the strengths and weaknesses of experimental research.
3. Diagnose and address threats to internal validity in experiments.
4. Analyze the results of an experiment.
5. Recognize the limitations of experiments and be able to thoughtfully critique them.
6. Design an experiment that you could implement and/or submit for funding.

What kind of student does this course have in mind?

This course is limited to doctoral students in Sociology and related disciplines. I will assume that you have taken at least an introductory statistics course (through linear regression). For a refresher, you may be interested in the following books:

Angrist, Joshua, and Steffan Jorg Pischke. 2009. Mostly Harmless Econometrics. Princeton: Princeton University Press.

Morgan, Stephen L., and Christopher Winship. 2014. Counterfactuals and Causal Inference: Methods and Principles for Social Research, Second Edition. Cambridge, UK: Cambridge University Press.

How will performance be assessed?

Assignment	Description	Deadlines
Short writing assignments (30%)	There are seven short assignments that build toward the final experiment proposal. Details about the assignments are provided along with the readings below. Assignments are graded for completeness.	Due throughout the semester via upload to Canvas by 11:59 pm the day before our Friday class meets.
Analysis paper (10%)	Based on data from an experiment, I will ask you answer a specific set of questions. More details will be provided in class.	Due via upload to Canvas before class begins on November 19.
Experiment proposal (40%)	The final assignment of the course is to develop a proposal for an experiment, ideally one that can be submitted for funding. Accordingly, I expect final assignments to be modelled after a TESS proposal ( http://tessexperiments.org/). If other formats are preferable, please let me know by email as soon as possible.	Due via upload to Canvas by 11:59pm on Monday, December 13.
In class participation (20%)	A key component of this seminar will be in-class student participation. Participation refers to informed contributions to the class. You should also be ready to discuss your short assignments in class. In addition, you will be asked to prepare a final presentation of your experimental proposal.

Please submit all assignments in 12-point type size, double-spaced, with one-inch margins. These are disciplinary standards and using double-space allows me to give better feedback. Please name all assignment submissions with your last name and first initial (e.g. chuj) followed by an underscore and a descriptive title of the assignment. All assignments should be uploaded to Courseworks. Following these guidelines will help me keep track of your assignments and focus on providing useful feedback.

What are the basic norms in this course?

• If you need disability-related accommodations, let me know as soon as possible. You have the right to have your needs met.
• If you need accommodations, you should be registered with the Office of Disability Services (ODS) in 008 Milbank (212-854-2388, disability@columbia.edu).
• Life happens.If you submit your work after the deadlines listed above, you will still qualify for half of the original points.
• Avoid using cell phones in class, which can prevent you or others from learning. In cases of emergencies, please take your phone outside.
• I try to respond to emails within 24 hours. You are welcome to follow-up only if I have not responded by then.

What are expectations regarding academic integrity?

Graduate students are expected to exhibit the highest level of personal and academic honesty as they engage in scholarly discourse and research. In practical terms, you must be responsible for the full and accurate attribution of the ideas of others in all of your research papers and projects; you must be honest when taking your examinations; you must always submit your own work and not that of another student, scholar, or internet source. Graduate students are responsible for knowing and correctly utilizing bibliographical guidelines.

Where can I access course materials?

The readings for each week of this course are listed below. There is one required textbook for this course:

Gerber, Alan S., and Donald P. Green. 2012. "Field Experiments: Design, Analysis, and Interpretation." 1st ed. New York: W.W. Norton.

Other than the required textbook, all other readings will be available on the course website on Courseworks.

Readings and Due Dates for Assignments

Part I: Understanding Experiments

Week 1: Why Experiment? (Sep 10)

Jackson, Michelle, and D. R. Cox. 2013. "The Principles of Experimental Design and Their Application in Sociology." Annual Review of Sociology 39:27–49.

Pearl, Judea and Dana Mackenzie. 2018. The Book of Why: The New Science of Cause and Effect. New York: Basic Books. Chapter 1.

http://bayes.cs.ucla.edu/WHY/why-ch1.pdf

Week 2: The Logic and Assumptions of Experiments (Sep 17)

Due: Submit at most 2 paragraphs summarizing an experiment that you want to develop in this course. At minimum, your summary should include a research question, why the question is important, and a rough sketch of how you plan to answer the question.

Druckman, James N. et al. 2011. "Cambridge Handbook of Experimental Political Science." Cambridge Handbook of Experimental Political Science.

Ch. 2, "Experiments: An Introduction to Core Concepts"

Gerber, Alan S., and Donald P. Green. 2012. "Field Experiments: Design, Analysis, and Interpretation." 1st ed. New York: W.W. Norton.

Ch. 1, "Introduction"

Ch. 2, "Causal Inference and Experimentation"

Week 3: Types of Experiments Pt I (Sep 24)

Lab Experiments

Willer, Robb. 2009. "Groups Reward Individual Sacrifice: The Status Solution to the Collective Action Problem." American Sociological Review 74(1):23-43.

Ridgeway, Cecilia L., and Shelley J. Correll. 2006. "Consensus and the Creation of Status Beliefs." Social Forces 85(1): 431-453.

Survey Experiments

Schachter, Ariela. 2016. "From 'Different' to 'Similar': An Experimental Approach to Understanding Assimilation." American Sociological Review 81(5):981-1013.

Phelan, Jo C., Bruce G. Link, and Naumi M. Feldman. 2013. "The Genomic Revolution and Beliefs about Essential Racial Differences: A Backdoor to Eugenics?" American Sociological Review 78(2):167-191.

Audit Experiments

Pager, Devah. 2003. "The Mark of a Criminal Record." American Journal of Sociology 108(5):937-975. doi: 10.1086/374403.

Correll, Shelley J., Stephen Benard, and In Paik. 2007. "Getting a Job: Is There a Motherhood Penalty?" American Journal of Sociology 112(5):1297-1339.

Week 4: Types of Experiments Pt II (Oct 1)

Due : Summarizing experiments memo (see handout)

Web Experiments

Salganik, Matthew J., Peter Sheridan Dodds, and Duncan J. Watts. 2006. "Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market." Science 311(854):854-856.

Kramer, A. D., Guillory, J. E., & Hancock, J. T. (2014). Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, 111(24), 8788-8790.

Field Experiments

Paluck, Elizabeth Levy, Hana Shepherd, and Peter M. Aronow. 2016. "Changing Climates of Conflict: A Social Network Experiment in 56 Schools." PNAS 113(3):556-571.

Broockman, David, and Joshua Kalla. 2016. "Durably Reducing Transphobia: A Field Experiment on Door-to-Door Canvassing." Science 352(6282):220-224.

Lab in the Field

Gneezy, Uri, and Imas, Alex. 2017. "Lab in the Field: Measuring Preferences in the Wild." Pp. 439-464 in Handbook of Economic Field Experiments, edited by A. Vinayak Banerjee and E. Duflo. North-Holland.

Baldassarri, Delia. 2015. "Cooperative networks: Altruism, group solidarity, reciprocity, and sanctioning in Ugandan producer organizations." American Journal of Sociology 121(2): 355-395.

Week 5: Hallmarks of Valid and Publishable Social Science Experiments (Oct 8)

Due : Write a title and abstract for a paper you imagine writing based on your proposed experiment. Assume that your findings align with your theoretical predictions. Remember to establish why the findings matter for your intended audience.

Druckman, James N. 2021. Experimental Thinking: A Primer on Social Science Experiments. New York: Cambridge University Press.

Ch. 3 "Evaluating Experiments: Realism, Validity, and Samples"

Ch. 6 "Designing "Good" Experiments"

McDermott, Rose. 2011. "Chapter 3. Internal and External Validity." in Druckman, James N ed. Cambridge Handbook of Experimental Political Science. Cambridge: Cambridge University Press.

Part II: Designing Experiments

Week 6: Treatment Assignment (Oct 15)

Due: Outline of your proposed experiment (see handout).

Familiarize yourself with the different types of research design in the following two links:

Declare Design - Descriptive Experimental Designs

Declare Design - Causal Experimental Designs

Charness, Gary, Uri Gneezy, and Michael A. Kuhn. 2012. "Experimental Methods: Between- Subject and Within-Subject Design." Journal of Economic Behavior & Organization 81:1-8.

Week 7: The Causal Chain (Oct 22)

Due: Revised outline, now including a new section called "Mediators and Moderators." In this section, describe at least one theorized mediator and moderator and how you plan to measure them. Include a proposed causal diagram that relates all independent and dependent variables and moderators and mediators.

Gerber, Alan S., and Donald P. Green. 2012. "Field Experiments: Design, Analysis, and Interpretation." 1st ed. New York: W.W. Norton.

Ch. 10, "Mediation"

Imai, Kosuke, Luke Keele, Dustin Tingley, and Teppei Yamamoto. 2011. "Unpacking the Black Box of Causality: Learning about Causal Mechanisms from Experimental and Observational Studies." American Political Science Review 105(4):765–89. doi: 10.1017/S0003055411000414.

Week 8: False Positives and Negatives (Oct 29)

Lee, Stephanie M. 2018. "Sliced & Diced: The Inside Story of How an Ivy League Food Scientist Turned Shoddy Data into Viral Studies." Buzzfeed.

Van Bavel, Jay. 2016, May 27. "Why Do So Many Studies Fail to Replicate?" The New York Times.

DeHaven, Alexander. 2017, June 23. "Preregistration: A Plan, Not a Prison." Center for Open Science. https://www.cos.io/blog/preregistration-plan-not-prison

Sullivan, Ian. 2020, August 13. "Preregistration." Open Science Framework Guides. https://help.osf.io/hc/en-us/articles/360021390833-Preregistration

Clifford, Scott, Geoffrey Sheagley, and Spencer Piston. 2021. "Increasing Precision without Altering Treatment Effects: Repeated Measures Designs in Survey Experiments." American Political Science Review 1–18.

Week 9: Threats to Internal Validity (Nov 5)

Due: Revised outline, now including a new section titled "Potential Threats." In this section, diagnose threats and briefly describe potential countermeasures. This new section should discuss false positives, statistical power, demand effects, noncompliance, spillover, and attrition.

Mummolo, Jonathan, and Erik Peterson. 2019. "Demand Effects in Survey Experiments: An Empirical Assessment." American Political Science Review 113(2):517–29. doi: 10.1017/S0003055418000837.

Kane, John V., and Jason Barabas. 2019. "No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments." American Journal of Political Science 63(1):234–49.

Gerber, Alan S., and Donald P. Green. 2012. "Field Experiments: Design, Analysis, and Interpretation." 1st ed. New York: W.W. Norton.

Ch. 5, "One-Sided Noncompliance"

Ch. 7, "Attrition"

Ch. 8, "Interference between Experimental Units"

Week 10: Ethics (Nov 12)

Due : Ethics Memo (see handout).

Be prepared to critique Stanford prison "experiment:"

Zimbardo, Phillip et al. 1971. The Stanford Prison Experiment: conducted August 1971 at Stanford University.

https://web.stanford.edu/dept/spec_coll/uarch/exhibits/Narration.pdf

McDermott, Rose, and Peter K. Hatemi. 2020. "Ethics in field experimentation: A call to establish new standards to protect the public from unwanted manipulation and real harms." Proceedings of the National Academy of Sciences 117(48): 30014-30021.

Matias, J. N. 2016, December 12. "The Obligation to Experiment." MIT Media Lab.

Week 11: Identifying and Interpreting Treatment Effects (Nov 19)

Due: Analysis memo (see handout)

Gerber, Alan S., and Donald P. Green. 2012. "Field Experiments: Design, Analysis, and Interpretation." 1st ed. New York: W.W. Norton.

Ch. 3, "Sampling Distributions, Statistical Inference, and Hypothesis Testing"

Mutz, Diana. 2011. Population-Based Survey Experiments. Princeton: Princeton University Press.

Ch. 7, "Analysis of Population-Based Survey Experiments"

NO CLASS NOV 26

Week 12: Final presentations (Dec 3)

Presentations are scheduled before the final week to allow time to integrate feedback to your proposal.

Week 13: New Advances and Critiques (Dec 10)

Guilbeault, D., Baronchelli, A. & Centola, D. 2021. Experimental evidence for scale-induced category convergence across populations. Nat Communications 12, 327.

Deaton, A., Cartwright, N. 2018. "Understanding and misunderstanding randomized

controlled trials." Social Science & Medicine 210: 2–21. Gelman. 2018. "Benefits and limitations of randomized controlled trials: A commen-

tary on Deaton and Cartwright." Social Science & Medicine 210: 48–49.

Tollefson, Jeff. 2015. "Can Randomized Trials Eliminate Global Poverty?" Nature 524(7564):150–53. doi: 10.1038/524150a.

Paluck, Elizabeth Levy. 2010. "The Promising Integration of Qualitative Methods and Field Experiments." The ANNALS 628(1):59-71.

Wager, Stefan, and Susan Athey. 2018. "Estimation and Inference of Heterogeneous Treatment Effects Using Random Forests." Journal of the American Statistical Association 113(523):1228–42. doi: 10.1080/01621459.2017.1319839.

Levine AS. 2020. "Research Impact Through Matchmaking (RITM): How and Why to Connect Researchers and Practitioners." PS: Political Science & Politics 53: 265-269_._

Al-Ubaydli, Amar, John A. List, and Dana L. Suskind. 2017. "What Can We Learn from Experiments? Understanding the Threats to the Scalability of Experimental Results." American Economic Review: Paper and Proceedings. 107(5): 282–286 https://doi.org/10.1257/aer.p20171115

Final Proposals Due (December 13)