Session 4 Hypothesis Testing

4.1 Core content

  • What is a good hypothesis?
  • What is the relationship between hypothesis testing and causal inference? Why test hypotheses when we want to learn about the causal effect of some intervention on some outcome?
  • What is a hypothesis test?
    • What is a null hypothesis?
    • What is a test statistic? When might we want to not use the difference of means as a test statistic? Estimators versus test statistics.
    • Where does the reference distribution for a hypothesis test come from? In an experiment, it comes from the experimental design and the randomization.
    • What is a \(p\)-value? How should we interpret the results of hypothesis tests?
  • What do we want from a hypothesis test?
    • A good test casts doubt on the truth rarely (i.e., has a controlled and low false positive rate)
    • A good test easily distinguishes signal from noise (i.e., casts doubt on falsehoods often; has high statistical power)
  • How would we know when our hypothesis test is doing a good job? (Power analysis is its own session)
    • What is a false positive rate? What is correct coverage of a confidence interval? (And why are we mentioning confidence intervals when we talk about hypothesis tests?)
    • How might we assess the false positive rate of a hypothesis test for a given design and choice of test statistic? (The case of cluster-randomized trials and robust cluster standard errors.)
  • Analyze as you randomize in the context of testing for causal inference in a randomized experiment means that we think of the reference distribution that generates our \(p\)-value as arising from repetitions of the randomization process of the experiment.

4.2 Slides

Below are slides with the core content that we cover in this session.

R Markdown Source

PDF Version

HTML Version

Slides from previous Learning Days

Hypothesis Testing Lecture as used in Learning Days 10 Bogota

4.5 Examples