--- title: "Design declaration, diagnosis, and redesign | *Déclaration, diagnostic et redesign*" subtitle: "MIDA — introduction | *MIDA — introduction*" author: "Macartan / Alyssa" date: today format: revealjs: embed-resources: true --- ```{r, include=FALSE} library(DeclareDesign) library(knitr) library(tidyverse) library(kableExtra) knitr::opts_chunk$set( echo = TRUE, warning = FALSE, error = TRUE, message = FALSE, eval = TRUE ) set.seed(343) theme_set(theme_light()) bs_style <- c("striped", "hover", "condensed", "responsive") options(kable_styling_bootstrap_options = bs_style) ``` ## Learning goals | *Objectifs d'apprentissage* ::: {.columns} ::: {.column .lang-en width="50%"} 1. Understand that if you can state the MIDA elements of a design in code, you can assess the quality of a design 2. Learn how design adjustments can be optimized 3. Get exposure to the kind of code that lets you do this in practice ::: ::: {.column .lang-fr width="50%"} 1. Comprendre que si vous pouvez déclarer les éléments MIDA d'un design en code, vous pouvez évaluer la qualité d'un design 2. Apprendre à optimiser les ajustements de design 3. Découvrir le type de code qui permet de le faire en pratique ::: ::: ## Declaration | *Déclaration* ::: {.columns} ::: {.column .lang-en width="50%"} **Declaration** tells the computer (and readers) what **M**, **I**, **D**, and **A** are. ::: ::: {.column .lang-fr width="50%"} La **déclaration** indique à l'ordinateur (et aux lecteurs) ce que sont **M**, **I**, **D** et **A**. ::: ::: ## Diagnosis | *Diagnostic* ::: {.columns} ::: {.column .lang-en width="50%"} **Diagnosis** asks how the design will perform under imagined conditions. We estimate **diagnosands**: power, bias, RMSE, error rates, ethical harm, amount learned, … ::: ::: {.column .lang-fr width="50%"} Le **diagnostic** examine comment le design se comportera dans des conditions imaginées. On estime des **indicateurs de diagnostic** : puissance statistique, biais, erreur quadratique moyenne, taux d'erreur, risques éthiques, quantité apprise, … ::: ::: ## Redesign | *Redesign* ::: {.columns} ::: {.column .lang-en width="50%"} **Redesign** adjusts data- and answer-strategy features to see how diagnosands change: - sample size - randomization procedure - estimation strategy - implementation trade-offs (e.g. compliance vs. sample size) ::: ::: {.column .lang-fr width="50%"} L'étape **redesign** (re-concevoir) adapte la stratégie de données et d'analyse afin d'observer comment les indicateurs de diagnostic évoluent. On peut modifier : - la taille de l'échantillon - la procédure d'assignation aléatoire (randomisation) - la stratégie d'analyse - les compromis liés à la mise en œuvre (p.ex. respect de la conformité vs. la taille d'échantillon) ::: ::: # Declaration | *Déclaration* ## Simplest design | *design la plus simple* ::: {.columns} ::: {.column .lang-en width="50%"} What is the simplest **diagnosable** design? ::: ::: {.column .lang-fr width="50%"} Quelle est la conception de recherche **diagnostiquable** la plus simple ? ::: ::: ```{r} simplest_design <- declare_model(N = 100, Y0 = rnorm(N), Y1 = Y0 + .2) + declare_inquiry(Q = mean(Y1- Y0)) + declare_assignment(X = simple_ra(N)) + declare_measurement(Y = X*Y1 + (1-X)*Y0) + declare_estimator(Y ~ X) ``` ## What it does | *Ce que fait le code* ::: {.columns} ::: {.column .lang-en width="50%"} - M: Draw 100 units with potential outcomes - I: Define an inquiry: the **sample average effect** - D: Assign a treatment and measure an outcome - A: Estimate the mean with a regression intercept ::: ::: {.column .lang-fr width="50%"} - M : Tirer 100 unités avec leurs résultats potentiels - I : Définir une interrogation (une question de recherche) : l'**effet moyen dans l'échantillon** - D : Assigner le traitement et mesurer le résultat - A : Estimer la moyenne avec l'ordonnée à l'origine d'une régression ::: ::: ## Run once | *Exécuter le code une fois* ::: {.columns} ::: {.column .lang-en width="50%"} Run the whole design once — type its name or call `run_design()`. This produces data, **estimands**, **estimates**, and ancillary statistics. ::: ::: {.column .lang-fr width="50%"} Exécutez le code de la conception de recherche dans son intégralité une fois — tapez son nom ou appelez `run_design()`. Cela génère des données, des **estimandes**, des **estimations** et des statistiques auxiliaires. ::: ::: ```{r, eval = FALSE} simplest_design |> run_design() ``` ```{r, echo = FALSE} one_run <- simplest_design |> run_design() one_run |> kable(digits = 2) |> kable_styling(font_size = 16) ``` ## Simulation | *Simulation* ::: {.columns} ::: {.column .lang-en width="50%"} Repeat the design many times with `simulate_design()`. ::: ::: {.column .lang-fr width="50%"} Répétez le design **plusieurs fois** avec `simulate_design()`. ::: ::: ```{r, include=FALSE, warning = FALSE} some_runs <- simplest_design |> simulate_design(sims = 100) ``` ```{r, warning = FALSE, eval = FALSE} some_runs |> head() ``` ```{r, warning = FALSE, echo = FALSE} some_runs |> head() |> kable(digits = 2) |> kable_styling(font_size = 14) ``` ## Distribution of estimates | *Distribution des estimations* ```{r, echo = FALSE} some_runs |> ggplot(aes(estimate)) + geom_histogram() ``` ## Distribution of errors | *Distribution des erreurs* ```{r, echo = FALSE} some_runs |> ggplot(aes(estimate - estimand)) + geom_histogram() ``` ## Distribution of p values | *Distribution des p-valeurs* ```{r, echo = FALSE} some_runs |> ggplot(aes(p.value)) + geom_histogram() ``` # Diagnosis | *Diagnostic* ## Bias by hand | *Biais à la main* ::: {.columns} ::: {.column .lang-en width="50%"} After many simulations, ask about **bias**: the average gap between estimand and estimate. ::: ::: {.column .lang-fr width="50%"} Après de nombreuses simulations, mesurez le **biais** : la différence moyenne entre l'estimande et l'estimation. ::: ::: ```{r, eval = FALSE} some_runs |> mutate(error = estimate - estimand) |> summarize( mean_estimate = mean(estimate), mean_estimand = mean(estimand), bias = mean(error) ) ``` ```{r, echo = FALSE} some_runs |> mutate(error = estimate - estimand) |> summarize( mean_estimate = mean(estimate), mean_estimand = mean(estimand), bias = mean(error) ) |> kable(digits = 2) |> kable_styling(font_size = 18) ``` ## diagnose_design() | *diagnose_design()* ::: {.columns} ::: {.column .lang-en width="50%"} `diagnose_design()` computes common diagnosands in one step. ::: ::: {.column .lang-fr width="50%"} `diagnose_design()` calcule les indicateurs de diagnostic en une seule étape. ::: ::: ```{r, eval = FALSE} diagnosis <- simplest_design |> diagnose_design() ``` ```{r, echo = FALSE} diagnosis <- simplest_design |> diagnose_design(sims = 1000) diagnosis |> reshape_diagnosis() |> select(-Inquiry, -Estimator, -Outcome, -Term) |> kable() |> kable_styling(font_size = 16) ``` # Redesign | *Redesign* ## What is redesign? | *Qu'est-ce que le redesign ?* ::: {.columns} ::: {.column .lang-en width="50%"} Make a new design by taking an existing design and change a **parameter** ```{r, eval = FALSE} design_2 <- design |> redesign(N = 300) ``` ::: ::: {.column .lang-fr width="50%"} Créez un nouveau design en partant d'un design existant et en modifiant une **variable** ```{r, eval = FALSE} design_2 <- design |> redesign(N = 300) ``` ::: ::: ## Design parameters | *Variables de design* {.smaller} ::: {.columns} ::: {.column .lang-en width="50%"} * To do this your design needs *parameters* * A **parameter** is a quantity in your environment that the design references explicitly — e.g. `N` instead of a fixed number. * You create it in your environment and then reference it when you make your design ::: ::: {.column .lang-fr width="50%"} * Pour cela, votre design a besoin de *variables* * Une **variable** est une quantité dans votre environnement que le design cite explicitement — p.ex. `N` au lieu d'un nombre fixé. * Vous la créez dans votre environnement, puis vous y faites référence lorsque vous construisez votre design. ::: ::: ```{r} N <- 100 simplest_design <- declare_model(N = N, Y0 = rnorm(N), Y1 = 1 + rnorm(N)) + declare_inquiry(Q = mean(Y1- Y0)) + declare_assignment(X = simple_ra(N)) + declare_measurement(Y = X*Y1 + (1-X)*Y0) + declare_estimator(Y ~ X) ``` ## Simple redesign | *Redesign simple* ::: {.columns} ::: {.column .lang-en width="50%"} `redesign()` returns a modified design — here with `N = 200`. ::: ::: {.column .lang-fr width="50%"} `redesign()` renvoie un design modifié — ici avec `N = 200`. ::: ::: ```{r} design_200 <- simplest_design |> redesign(N = 200) design_200 |> draw_data() |> nrow() ``` ## A list of designs | *Une liste de designs* {.smaller} ::: {.columns} ::: {.column .lang-en width="50%"} * Pass a **vector** of parameter values to get several designs at once. * With two parameters, the function `redesign()` builds a grid of designs — easy to diagnose and compare. ::: ::: {.column .lang-fr width="50%"} * Passez un **vecteur** de valeurs de variables pour obtenir plusieurs designs à la fois. * Avec deux variables, la fonction `redesign()` construit un tableau de designs — facile à diagnostiquer et à comparer. ::: ::: ```{r} N <- 100 b <- .2 simplest_design <- declare_model(N = N, Y0 = rnorm(N), Y1 = b + rnorm(N)) + declare_inquiry(Q = mean(Y1- Y0)) + declare_assignment(X = simple_ra(N)) + declare_measurement(Y = X*Y1 + (1-X)*Y0) + declare_estimator(Y ~ X) ``` ```{r, eval = FALSE} designs <- redesign(simplest_design, N = c(100, 200, 500), b = c(0, .2, .5)) designs |> diagnose_design() |> tidy() ``` ## Compare diagnoses | *Comparer les diagnostics* ```{r, echo = FALSE} designs <- redesign(simplest_design, N = c(100, 200, 500), b = c(0, .2, .5)) redesign_tidy <- designs |> diagnose_design(sims = 250) |> tidy() redesign_tidy |> select(N, b, diagnosand, estimate, std.error, conf.low, conf.high) |> filter(diagnosand %in% c("power", "rmse")) |> kable(digits = 2) |> kable_styling(font_size = 14) ``` ## Plot after redesign | *Graphique après redesign* {.smaller} ::: {.columns} ::: {.column .lang-en width="50%"} After `tidy()`, ggplot makes comparison across redesigns straightforward. ::: ::: {.column .lang-fr width="50%"} Après `tidy()`, ggplot permet de comparer facilement les redesigns. ::: ::: ```{r, echo = FALSE, fig.height = 2.2, fig.width = 6} redesign_tidy |> filter(diagnosand %in% c("power", "rmse")) |> ggplot(aes(N, estimate, color = factor(b))) + geom_line() + facet_wrap(~diagnosand) ``` ::: {.columns} ::: {.column .lang-en width="50%"} Power depends on *N* and effect size; RMSE depends on *N* only. ::: ::: {.column .lang-fr width="50%"} La puissance statistique dépend du *N* et de la taille de l'effet ; L'erreur quadratique moyenne ne dépend que de *N*. ::: ::: # Resources | *Ressources* ## declare_* commands | *Commandes declare_* ::: {.columns} ::: {.column .lang-en width="50%"} Key steps for **building** a design: - `declare_model()` - `declare_inquiry()` - `declare_sampling()` - `declare_assignment()` - `declare_measurement()` - `declare_estimator()` - … and more `declare_*` functions ::: ::: {.column .lang-fr width="50%"} Étapes clés pour **construire** un design : - `declare_model()` - `declare_inquiry()` - `declare_sampling()` - `declare_assignment()` - `declare_measurement()` - `declare_estimator()` - … et d'autres fonctions `declare_*` ::: ::: ## Using a design | *Utiliser un design* ::: {.columns} ::: {.column .lang-en width="50%"} Key commands for **running** and **learning** from a design: - `draw_data(design)`, `draw_estimands(design)`, `draw_estimates(design)` - `get_estimates(design, data)` - `run_design(design)`, `simulate_design(design)` - `diagnose_design(design)` - `redesign(design, N = 200)` - `compare_designs()`, `compare_diagnoses()` ::: ::: {.column .lang-fr width="50%"} Commandes clés pour **exécuter** un design et **en tirer des conclusions** : - `draw_data(design)`, `draw_estimands(design)`, `draw_estimates(design)` - `get_estimates(design, data)` - `run_design(design)`, `simulate_design(design)` - `diagnose_design(design)` - `redesign(design, N = 200)` - `compare_designs()`, `compare_diagnoses()` ::: ::: ## Other resources | *Autres ressources* ::: {.columns} ::: {.column .lang-en width="50%"} - Full slide deck: [macartan.github.io/dd_bootcamp](https://macartan.github.io/dd_bootcamp/) - Website: [declaredesign.org](https://declaredesign.org/) - Book: [book.declaredesign.org](https://book.declaredesign.org) - Console help: `?DeclareDesign` ::: ::: {.column .lang-fr width="50%"} - Diapositives complètes : [macartan.github.io/dd_bootcamp](https://macartan.github.io/dd_bootcamp/) - Site web : [declaredesign.org](https://declaredesign.org/) - Livre : [book.declaredesign.org](https://book.declaredesign.org) - Aide console : `?DeclareDesign` ::: :::