Module 6 Estimands and Estimators

Randomized experiments generate good guesses about the average outcome under treatment and the average outcome under control. This allows us to write down unbiased estimators of average treatment effects. We can also use the randomization to describe how estimates generated by an estimator can vary from experiment to experiment in the form of standard errors and confidence intervals.

In this module, we introduce several types of estimands, the target quantity to be estimated. The choice of estimand is a scientific and policy-informed decision – what quantity is useful for us to learn about? In addition, we want to select an appropriate estimator for this estimand as part of the research design. We discuss how estimators are applied to data to generate an estimate of our estimand and how to characterize the variability of this estimate.

6.1 Core Content

  • A causal effect, \(\tau_i\), is a comparison of unobserved potential outcomes for each unit \(i\). For example, this can be a difference or a ratio of unobserved potential outcomes.

  • To learn about \(\tau_{i}\), we can treat \(\tau_{i}\) as an estimand or target quantity to be estimated (this module) or as a target quantity to be hypothesized about (hypothesis testing module).

  • Many focus on the average treatment effect (ATE), \(\bar{\tau}=\sum_{i=1}^n \tau_{i}\), in part, because it allows for easy estimation.

  • An estimator is a recipe for calculating a guess about the value of an estimand. For example, the difference between the mean of observed outcomes for \(m\) treated units and the mean of observed outcomes for \(N-m\) untreated units is one estimator of \(\bar{\tau}\).

  • Different randomizations will produce different values of the same estimator targeting the same estimand. A standard error summarizes this variability in an estimator.

  • A \(100(1-\alpha)\)% confidence interval is a collection of hypotheses that cannot be rejected at the \(\alpha\) level. We tend to report confidence intervals containing hypotheses about values of our estimand and use our estimator as a test statistic.

  • Estimators should (1) avoid systematic error in their guessing of the estimand (be unbiased); (2) vary little in their guesses from experiment to experiment (be precise or efficient); and perhaps ideally (3) converge to the estimand as they use more and more information (be consistent).

  • Analyze as you randomize in the context of estimation means that (1) our standard errors should measure the variability from randomization and (2) our estimators should target estimands defined in terms of potential outcomes.

  • We do not control for background covariates when we analyze data from randomized experiments. But covariates can make our estimation more precise. This is called covariance adjustment. Covariance adjustment in randomized experiments differs from controlling for variables in observational studies.

  • A policy intervention (like a letter that encourages exercise) may intend to change behavior via an active dose (actual exercise). We can learn about the causal effect of the intention by randomly assigning letters; this is the intent to treat effect, ITT.

  • We can learn about the causal effect of actual exercise by using the random assignment of letters as an instrument for the active dose (exercise itself) in order to learn about the causal effect of exercise among those who would change their behavior after receiving the letter. The average causal effect versions of these effects are often known as the complier average causal effect or the local average treatment effect.

6.3 Resources

6.3.2 Books, Chapters, and Articles

  • Gerber and Green, Field Experiments. Chapter 2.7 on excludability and non-interference, Chapter 3, Chapter 5 on one-sided noncompliance, Chapter 6 on two-sided noncompliance, Chapter 7 on attrition, Chapter 8 on interference between experimental units.

  • Jake Bowers and Thomas Leavitt, “Causality & Design-Based Inference,” in The Sage Handbook of Research Methods in Political Science and International Relations, ed. Luigi Curini and Robert Franzese (Sage Publications Ltd, 2020).