Session 9 Glossary of Terms

Below are some core terms frequently used.

9.1 Key Concepts

  • Potential outcome \(Y_i(Z)\) What outcome \(Y\) that unit \(i\) have under treatment condition \(Z\). We think of these as fixed quantities. Z can be 0 (for control) or 1 (for treatment)
  • Treatment effect \(\tau_i\) for unit \(i\) The difference between potential outcomes under treatment and control, \(Y_i(1)-Y_i(0)\).
  • Fundamental Problem We can’t observe \(Y_i(1)\) and \(Y_i(0)\) for the same unit at the same time, so we can’t get \(\tau_i\) directly.
  • Average treatment effect, ATE The average of the treatment effect for all individuals in your subject pool. \(\overline{Y_i(1)-Y_i(0)}\), which is also equivalent to \(\bar{Y}_i(1)-\bar{Y}_i(0)\). Notice that we do not use the \(E[Y_i (1)]\) style of notation here because \(E[]\) means “average over repeated operations” but \(\bar{Y}\) means “average over a set of observations”.
  • Estimand The thing you want to estimate. Example: average treatment effect.
  • Estimator How you estimate your estimand from the data you have. Example: difference-in-means.
  • Unbiased An estimator is unbiased if you expect that it will return the right outcome. That means that if you were to run the experiment many times, the estimate might be too high or to low sometimes but it will be right on average.
  • Random Sampling Selecting subjects from a population with known probabilities (strictly between 0 and 1).
  • \(k\)-Arm Experiment An experiment that has \(k\) treatment conditions (including control).
  • Random Assignment Assigning subjects with known probability (without replacement) strictly between 0 and 1 to experimental conditions. This is the same as random sampling from the potential outcomes. There are several strategies for random assignment: simple, complete, cluster, block, blocked cluster.
  • External Validity Findings from your study are valid outside of your sample in other locations or in other interventions.

9.2 Statistical Inference

  • Hypothesis Simple, clear, falsifiable claim about the world.
  • Null Hypothesis A conjecture about the world that you may reject after seeing the data.
  • Sharp Null Hypothesis of No Effect The null hypothesis that there is no treatment effect for any subject. This means \(Y_i(1)=Y_i(0)\) for all \(i\).
  • \(p\)-value The probability seeing a test statistic as large (in absolute value) as the observed test statistic (e.g., the difference in means)
  • One-sided vs.~Two-sided Test When you have a strong expectation that the effect is either positive or negative, you can conduct a one-sided test. When you do not have such a strong expectation, conduct a two-sided test. A one-sided test has more power than a two-sided test for the same experiment.
  • Sampling Distribution The distribution of estimates (e.g., estimates of the ATE) for all possible treatment assignments.
  • Standard Deviation Square root of the mean-square deviation from average of a variable. It is a measure of the dispersion or spread of a statistic. \(SD_x=\sqrt{\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2}\)
  • Standard Error The standard deviation of the sampling distribution. A bigger standard error means that our estimates are more susceptible to sampling variation.
  • Statistical Power Probability that our experiment will detect a statistically significant treatment effect if the effect exists. This depends on:
    • The number of observations in each arm of the experiment
    • Effect size
    • Noisiness of the outcome variable
    • Significance level (\(\alpha\), which is fixed by convention)
    • Other factors including what proportion of your units are assigned to different treatment conditions
  • Intra-Cluster Correlation How correlated the potential outcomes of units are within clusters compared to across clusters. Higher intra-cluster correlation hurts power.

9.3 Randomization Strategies

  • Simple An independent coin flip for each unit. You are not guaranteed that your experiment will have a specific number of treated units.
  • Complete Assign \(m\) out of \(N\) units to treatment, i.e., you know how many units will be treated in your experiment. Each unit has a \(m/N\) probability of being treated. The number of ways treatment can be assigned (number of permutations of treatment assignment) is \(\frac{N!}{m!(N-m)!}\).
  • Block First divide the sample into blocks, then complete randomization in each block separately. A block is a set of units within which you conduct random assignment.
  • Cluster Clusters of units are randomly assigned to treatment conditions. A cluster is a set of units that will always be assigned to the same treatment status.
  • Blocked Cluster First form blocks of clusters. Then in each block, randomly assign the clusters to treatment conditions using complete randomization.

9.4 Factorial Designs

  • Factorial Design A design with more than one treatment, with each treatment assigned independently. The simplest factorial design is a 2 by 2.
  • Conditional Marginal Effect The effect of one treatment, conditional on the other being held at a fixed value. For example: \(Y_i(Z_1=1|Z_2=0)-Y_i(Z_1=0|Z_2=0)\) is the marginal effect of \(Z_1\) conditional on \(Z_2=0\).
  • Average Marginal Effect Main effect of each treatment in a factorial design. It is the average of the conditional marginal effects for all the conditions of the other treatment, weighted by the proportion of the sample that was assigned to each condition.
  • Interaction Effect In a factorial design, we may also estimate interaction effects.
    • No interaction effect: one treatment does not amplify or undercut the effect of the other treatment.
    • Multiplicative interaction effect: the effect of one treatment depends on whether a unit was assigned the other treatment. This means one treatment amplify or undercut the effect of the other. The effect of two treatments together is the sum of the effect of each treatment.

9.5 Threats

  • Hawthorne Effects When a subject responds to being observed.
  • Spillovers When a subject responds to another subject’s treatment status. Example: my health depends on whether my neighbor is vaccinated, as well as whether I am vaccinated.
  • Attrition When outcomes for some subjects are not measured. Example: people migrate or people die. This is especially problematic for inference when correlated with treatment status.
  • Compliance A unit’s treatment status matches its assigned treatment condition. Example of non-compliance: a unit assigned to treatment doesn’t take it. Example of compliance: a unit assigned to control does not take treatment.
  • Compliance Types There are four types of units in terms of compliance:
    • Compliers Units that would take treatment if assigned to treatment and would be untreated if assigned to control.
    • Always-Takers Units that would take treatment if assigned to treatment and if assigned to control.
    • Never-Takers Units that would be untreated if assigned to treatment and if assigned to control.
    • Defiers Units that would be untreated if assigned to treatment and would take treatment if assigned to control.
  • One-sided Non-Compliance The experiment has only compliers and {} always takers or never takers. Usually, we think of one-sided non-compliance as having only never takers and compliers meaning that that local average treatment effect is the effect of treatment on the treated.
  • Two-sided Non-Compliance The experiment may have all four latent groups.
  • Encouragement Design An experiment that randomizes \(Z\) (treatment assignment), and we measure \(D\) (whether the unit takes treatment) and \(Y\) (outcome). We can estimate the ITT and the LATE (Local Average Treatment Effect, aka CACE—Complier Average Causal Effect). It requires three assumptions.
    • Monotonicity Assumption of either no defiers or no compliers. Usually we assume no defiers which means that the effect of assignment on take up of treatment is either positive or zero but not negative.
    • First Stage Assumption that there is an effect of \(Z\) on \(D\).
    • Exclusion Restriction Assumption that \(Z\) affects \(Y\) only through \(D\). This is usually the most problematic assumption.
  • Intention-to-Treat Effect (ITT) The effect of \(Z\) (treatment assignment) on \(Y\).
  • Local Average Treatment Effect (LATE) The effect of \(D\) (taking treatment) on \(Y\) for compliers. Also known as Complier Average Causal Effect (CACE). Under the exclusion restriction and monotonicity, the LATE is equal to ITT divided by the proportion of your sample who are Compliers.
  • Downstream Experiment An encouragement design study that takes advantage of the randomization of \(Z\) by a previous study. The outcome from that previous study is the \(D\) in the downstream experiment.