Commentary and Analysis of “Field Experiments and the Practice of Policy” by Esther Duflo (American Economic Review, 2020)
Dr. Duflo critiques a simplistic view of how researchers influence policy through RTCs: conducting a small experiment, obtaining positive results, preparing a policy brief, and achieving full-scale adoption. This "simplistic bad practice" fuels criticisms such as context dependency and selection bias. To counter this view, she explains that RCT researchers do not draw conclusions from single experiments. By showcasing examples from studies that initially appeared to "fail" and discussing scalability approaches, she demonstrates that moving from proof-of-concept to scalable policy requires many experiments and co-creation between researchers and policymakers. For example, evidence from microcredit studies shifted policy debates from viewing microcredit as a miracle solution to a more nuanced understanding, leading to improved microfinance practices tailored to diverse borrower needs. Additionally, the TARL approach successfully scaled up in India and Africa, significantly improving learning outcomes. Finally, she emphasizes that implementing scalable policies from proof-of-concept necessitates numerous experiments and close collaboration between researchers and governments.
Dr. Duflo provides strong evidence for her claims about the use of RCT insights and offers actionable steps for collaborating with policymakers. Her perspective on leveraging RCTs beyond just "positive results" is supported by other work, such as that of Osman et al. (2020), who further support that examining interventions that fail provides valuable insights, especially by identifying the causal pathways of different types of failure. Osman et al demonstrate that a taxonomy of these interactions can advance both theory and practice in behavioral change initiatives.
However, while Dr. Duflo argues that implementing scalable policies from proof-of-concept requires many experiments and close collaboration between researchers and governments, scaling social interventions often leads to a "voltage drop," reducing effectiveness in larger, diverse settings (List, 2024). To mitigate this, List contends that we must optimally generate policy-based evidence by determining what information to produce and in what sequence. This may involve reversing traditional research models to include scalability tests from the outset, an argument that Dr. Duflo does not fully address in her paper.
Dr. Duflo emphasizes that moving from proof-of-concept to scalable policy requires iterative experimentation, co-creation with stakeholders, and tailored interventions based on robust evidence. Similarly, the study on reducing conspiracy theory beliefs by Costello, Pennycook and Rand (Science 2024) demonstrates the effectiveness of personalized, evidence-based dialogues (developed through advances in AI) to address beliefs. Both views highlight that generic solutions often fail to produce significant impact when scaled up due to varying individual needs and other complex factors. By leveraging tailored approaches and engaging directly with the target audience's specific concerns, both the researchers in the conspiracy study and those following Dr. Duflo's methodology shows that significant change is achievable. The alignment between these views further supports the importance of combining RCT testing, practical implementation, and collaboration to design interventions that are not only effective in controlled settings but also scalable and impactful in diverse real-world environments.
Considering the importance of collaboration between scientists and policymakers, is the effort currently reciprocal? Are scientists contributing more than policymakers to address these issues, and how can any imbalance be addressed?
References:
“Field Experiments and the practice of policy” - Esther Duflo - AER 2020
“Learning from behavioral changes that fail” - Osman, McLachlan, Fenton, Neil, Lofs - Trends in Cognitive Sciences 2020
“Optimally generate policy-based evidence before scaling” - John A. List - Nature 2024
“Durably reducing conspiracy beliefs through dialogues with AI” - Costello, Pennycook, Rand - Science 2024
