Module 9 Threats to the Internal Validity of Randomized Experiments

Randomized experiments can run into issues that undermine their ability to demonstrate causal effects – that is, threaten the internal validity of randomized experiments. Some units might be missing outcome data and that missingness may be due to the treatment. They may not take the treatment status assigned to them or be subject to spillover effects from a treated neighbor.

In this module, we cover some common threats and some best practices to avoid or work around them.

9.1 Core Content

  • Review the three core assumptions discussed in the causal inference module.

  • We have said “Analyze as you randomize” in the module on estimands and estimators. Remember that you randomized treatment assignment, not whether the treatment is received or whether a unit participates in data collection.

  • Missing data on the outcome (attrition) is especially a problem if the patterns of missingness are caused by the treatment itself. This is a very common problem.

    • Do not drop observations that are missing outcome data from your analysis.

    • You may be able to bound estimates of treatment effects.

  • Non-compliance. The effect of treatment assignment is not the same as the effect of receiving the treatment. Sometimes units will not comply with their assigned treatment status.

    • One-sided compliance occurs when some units assigned to treatment fail to take the treatment, but all units assigned to control do not take the treatment.

    • The local average treatment effect (LATE, also known as the complier average causal effect, CACE) is the average effect for the units that take the treatment when assigned, but not otherwise. If the monotonicity assumption and the exclusion restriction hold, we may be able to estimate LATE when we have non-compliance.

  • “Spillover effects” or interference between units is a violation of one of the core assumptions for causal inference (causal inference).

    • However, this may not be a problem if you are interested in spillover effects and/or have designed your research to account for it.

  • Hawthorne effects are when subjects behave differently because they are being observed.

  • Non-excludability. Treating treatment and control units differently, such as with different data collection processes or extra attention to the treated units, can confuse interpretation of experimental results.

    • If Hawthorne effects are present for treated units but not control units, then we have a violation of the excludability assumption.

9.2 Slides

Below are slides with the core content that we cover in our lecture on threats to the internal validity of randomized experiments. You can directly use these slides or make your local copy and edit.

You can also see the slides used in previous EGAP Learning Days:

9.3 Resources

9.3.2 Books, Chapters, and Articles