Module 2 The Research Design Process

This book aims to help you understand and design randomized field experiments. But before we dive into the details of research design, we need a good research question – a question that will advance knowledge or help make a policy decision or both. There is no simple recipe for finding or developing a good scientific or policy question, but our theories are important for articulating good questions that underlie impactful research. After formulating our question, we develop the best design possible within our resource constraints, using our knowledge of causal inference and statistics from the modules that follow.

This module introduces the EGAP Research Design Form, a checklist to guide you through the many stages of the research process. The Learning Days workshops are organized around the Research Design Form. We also point you towards the DeclareDesign software package to explore the implications of different choices we could make for our research designs. Finally, this module discusses pre-analysis plans and pre-registration. When plan our analyses and make these plans public, we improve our chances of persuading others with our results.

2.1 Core Content

  • A good research question advances science and/or is a question the answer to which will inform a policy decision.

  • Certain research designs are better able to address certain questions. We want to choose the design that best answers our key questions within our constraints.

  • The questions we ask arise — often implicitly — from our values and from our understanding about how the world works. These theories make our questions relevant. And the experiments that we execute teach us about the theory. That is, we hope that the evidence and data arising from these research designs improve our understanding.

  • Core components of a research design

  • Introduce core components of the EGAP Research Design Form.

  • Introduce a research design software package, DeclareDesign.

  • The move in social science towards the review of designs, rather than outcomes.

  • Pre-registration – what it is, and why and how we should do it.

2.2 Slides

Below are slides with the core content that we cover in our lecture on research design. You can directly use these slides or make your own local copy and edit.

You can also see the slides used in previous EGAP Learning Days:

You can also see slides for Design Talks in previous EGAP Learning Days, where presenters focus on issues that come up in designing the research, rather than the results:

2.3 Design Form and Pre-registration

2.4 Resources

2.4.1 EGAP Methods Guides

2.4.2 Books, Chapters, and Articles

  • Preregistration as a Tool for Strengthening Federal Evaluation. A white paper from the US Government’s Office of Evaluation Sciences. You can also see examples of their pre-analysis plans on all of their field experiment pages.

  • Garret S. Christensen, Jeremy Freese, and Edward Miguel, Transparent and Reproducible Social Science Research: How to Do Open Science (Oakland, California: University of California Press, 2019). The book summarizes new approaches in social science research on transparency and reproducibility.

  • Alan S. Gerber and Donald P. Green, Field Experiments: Design, Analysis, and Interpretation (New York, NY: W. W. Norton & Company, 2012). Chapter 12 includes some examples of experimental research designs.

2.4.3 Tools

  • DeclareDesign, an exciting and comprehensive set of software tools for describing, assessing, and conducting empirical research.

References

Christensen, Garret S., Jeremy Freese, and Edward Miguel. Transparent and Reproducible Social Science Research: How to Do Open Science. Oakland, California: University of California Press, 2019.

Gerber, Alan S., and Donald P. Green. Field Experiments: Design, Analysis, and Interpretation. New York, NY: W. W. Norton & Company, 2012.