Replication
Due by 11:55 PM on Friday, March 22, 2024
For the final course project, you will produce a short research paper that uses regression analysis to investigate a causal relationship. This page provides guidance for those who decide to pursue a replication project and present extensions to an existing study. It also provides information on what to submit for Milestones 1 and 2, and how to submit your final documents.
In summary, for the replication project you will
- identify an experimental article in your field of study,
- reproduce the article’s tables and figures, and
- present three extensions to the original study.
You will work by yourself, not in groups. At the same time, do not hesitate to reach out to other classmates via Perusall. Also, absolutely do not hesitate to ask me questions. I’m here to help!
Your first milestone is due by 11:55 PM on Sunday, February 04, 2024. You should meet with me at least once, during my student hours, at any point before 05:00 PM on Wednesday, February 14, 2024. Your second milestone is due by 11:55 PM on Sunday, March 03, 2024. No late work will be accepted.
For Milestone 2 and for your final paper, you should submit a replication package I (or anyone else) can run on my computer, along with your paper (i.e., not just a copy of your paper). A key aim of our class is for you learn how to create reproducible work—this assignment provides an opportunity to show that you have mastered this skill.
Overall instructions
One of the first research papers I wrote was a replication assignment for Gary King’s methods class “Quantitative Social Science Methods, I”. I learned a lot from this assignment, and—full disclosure—I’m now largely basing this course assignment on his class.
Start by reading Gary’s article “Publication, Publication” (King 2006). Then, after reading the article, return to this page and go over my additional instructions. I have changed the assignment slightly, so you should eventually stick to my instructions here.
Here is how your assignment differs from what you just read about in “Publication, Publication.”
- You will work by yourself, not with a co-author.
- You must select an experimental study.
- The study must come from your field of study. If you are a PhD student at the School of Education, the study must focus on a research question that matches your area of study.
- You should include a directed acyclic graph (DAG) in your paper.
- Unless there is no baseline data collection round, your paper should include an analysis of differential attrition and baseline balance.
- You should not submit any hard copy materials; all assignments and submissions go through Canvas.
Milestone 1
By Milestone 1, you should have identified the study you would like to replicate, and you should have sourced the necessary data. Also, you should have developed a good understanding of all the analyses in the original article and how the paper is able to make causal claims.
Here’s a summary of what you’ll need to submit for your first milestone (on Canvas).
- A PDF copy of the paper you are replicating.
- A PDF with a two-pager (or shorter document) that includes (a) a brief paragraph explaining your choice, including an argument for why your article matches the criteria listed in “Publication, Publication”, (b) a confirmation that you have successfully sourced all the necessary data to replicate the article, and (c) a DAG that summarizes the causal relationship investigated in the study, including a discussion of which relevant “backdoors” are closed by random assignment.
1:1 meeting
You should meet with me at least once to discuss your project. Please meet with me after submitting your first milestone and before Wednesday, February 14, 2024 (incl.). Feel free to meet with me more often; I look forward to hearing about your work!
Milestone 2
For Milestone 2, you should replicate all of the tables and figures of the original paper. You should prepare a replication package you can share with others. Also, you should provide a draft of your paper that includes an explanation of the extensions you’re planning to implement so I can provide you with feedback.
Follow the instructions of “Publication, Publication” closely. For example, your paper should “[c]larify with precision the extent to which you were able to replicate the author’s results.”
Unless there is no baseline data collection round, your paper should include an analysis of differential attrition and baseline balance. These do not count as “extensions.”
With “extensions,” I mean improvements you would like to make. For example, these may include:
- An improved presentation of the article’s findings (e.g., presenting a results table as a coefficient plot).
- An analysis of whether, using data from the control group only, a regression-based approach yields the same results, using Oster’s method (Oster 2019).
- Robustness checks (e.g., to account for differential attrition).
- Improvements over the article’s measurement approach.
- Corrections to standard errors (e.g., if the original article did not properly account for multiple hypothesis testing).
- Additional subgroup analyses (accounting for multiple hypothesis testing).
If appropriate, please include estimating equations for the additional analyses you are proposing.
Here’s a summary of what you’ll need to submit for your second milestone (on Canvas).
- A PDF of your draft paper that includes (a) a replication of all the original tables and figures and (b) an explanation of the three extensions you’re planning to implement.
- In the week before Milestone 2, I will pair you with a classmate (your “replicator”). In the week before the due date, you should share a replication package of your analyses with this classmate. Your classmate should be able to replicate all of your work with a single push of a button. For Milestone 2, please submit a screenshot or PDF printout of an email from your replicator, confirming (a) that your code allows for a push-button reproduction of all tables and figures and, vice versa, (b) that you were able to reproduce your classmate’s findings.
- The same replication package (as a .zip file) which allows me to replicate all of your tables and figures with a single push of a button.
Please also post an abstract of your paper via Jamboard. Include what your extensions consist of (though you won’t have your findings yet). That way, everyone can see what you’re working on. Limit your abstract to 150 words (not 250).
Final submission
Your submission of the final paper should improve over Milestone 2 in at least two ways. First, it should present the results from your three extensions. Second, you will have received feedback from me, and you may follow some of my suggestions.
Please make sure your paper follows the APA standards (JARS-Quant)—even if the original article did not adhere to these standards. That said, do not follow the “reporting standards for replication studies”; to the best of my understanding, they refer to replications of studies with a new sample and new data.
Here’s a summary of what you’ll need to submit for your final assignment (on Canvas).
- A PDF of your final paper, including the three extensions you have implemented.
- A replication package (as a .zip file) which allows me to replicate all of your tables and figures with a single push of a button.