IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v48y2024i4p579-609.html

Randomized Controlled Trial Aversion among Public Sector Leadership: A Survey Experiment

Author

Listed:
  • Emily Cardon
  • Leonard Lopoo

Abstract

Background: While randomized controlled trials (RCTs) are typically considered the gold standard of program evaluation, they are infrequently chosen by public sector leaders, defined as government and nonprofit decision-makers, when an impact evaluation is required. Objectives : This study provides descriptive evidence on RCT aversion among public sector leaders and attempts to understand what factors affect their likelihood of choosing RCTs for impact evaluations. Research Design : The authors ask if public sector leaders follow similar preference patterns found among non-public sector leaders when choosing either an RCT or a quasi-experimental design and use a survey experiment to determine which factors affect the RCT choice. Subjects : The study sample includes 2050 public sector leaders and a comparison group of 2060 respondents who do not lead public sector organizations. Measures: The primary outcome measure is selecting an RCT as the preferred evaluation option. Results : When asked to make a decision about an impact evaluation, the majority of people do not choose an RCT. While also averse to RCTs, public sector leaders are about 13% more likely to prefer a RCT to a quasi-experimental evaluation compared to the general population. Public sector leaders are less likely to use RCTs for evaluations of more intense interventions, potentially because they are perceived to be superior to the options available for the control group. Conclusion : Funders should be aware that when given a choice, public sector leaders prefer other options to RCTs. Greater awareness of the benefits of RCTs could increase their use in the public sector.

Suggested Citation

  • Emily Cardon & Leonard Lopoo, 2024. "Randomized Controlled Trial Aversion among Public Sector Leadership: A Survey Experiment," Evaluation Review, , vol. 48(4), pages 579-609, August.
    • Handle: RePEc:sae:evarev:v:48:y:2024:i:4:p:579-609
    DOI: 10.1177/0193841X231193483
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X231193483
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X231193483?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Rachel Glennerster & Kudzai Takavarasha, 2013. "Running Randomized Evaluations: A Practical Guide," Economics Books, Princeton University Press, edition 1, number 10085, December.
    2. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    3. Weinstein, N.D. & Klotz, M.L. & Sandman, P.M., 1988. "Optimistic biases in public perceptions of the risk from radon," American Journal of Public Health, American Public Health Association, vol. 78(7), pages 796-800.
    4. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    5. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Florian Spitzer & Kira Abstiens & Sophie Karmasin, 2025. "Integrating behavioural insights in the policy process: on chances and hurdles identified by policy-makers and behavioural scientists," Mind & Society: Cognitive Studies in Economics and Social Sciences, Springer;Fondazione Rosselli, vol. 24(2), pages 621-663, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christopher J. Ruhm, 2019. "Shackling the Identification Police?," Southern Economic Journal, John Wiley & Sons, vol. 85(4), pages 1016-1026, April.
    2. Vellore Arthi & James Fenske, 2018. "Polygamy and child mortality: Historical and modern evidence from Nigeria’s Igbo," Review of Economics of the Household, Springer, vol. 16(1), pages 97-141, March.
    3. Ravallion, Martin, 2020. "Highly prized experiments," World Development, Elsevier, vol. 127(C).
    4. repec:wbk:wbrwps:10250 is not listed on IDEAS
    5. Vikram Dayal & Anand Murugesan, 2020. "Demystifying causal inference: ingredients of a recipe," IEG Working Papers 393, Institute of Economic Growth.
    6. John Cawley & Alex Susskind & Barton Willage, 2020. "The Impact of Information Disclosure on Consumer Behavior: Evidence from a Randomized Field Experiment of Calorie Labels on Restaurant Menus," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(4), pages 1020-1042, September.
    7. Reimer, Matthew N. & Haynie, Alan C., 2018. "Mechanisms matter for evaluating the economic impacts of marine reserves," Journal of Environmental Economics and Management, Elsevier, vol. 88(C), pages 427-446.
    8. Florian F Gunsilius, 2025. "A primer on optimal transport for causal inference with observational data," Papers 2503.07811, arXiv.org, revised Mar 2025.
    9. Dina Pomeranz, 2017. "Impact Evaluation Methods in Public Economics," Public Finance Review, , vol. 45(1), pages 10-43, January.
    10. Jeffery A. Jenkins & Nolan McCarty & Charles Stewart, 2020. "Learning from each other: causal inference and American political development," Public Choice, Springer, vol. 185(3), pages 245-251, December.
    11. Jonathan Bauchet & Stacy Prieto & Jacob Ricker‐Gilbert, 2021. "Improved Drying and Storage Practices that Reduce Aflatoxins in Stored Maize: Experimental Evidence from Smallholders in Senegal," American Journal of Agricultural Economics, John Wiley & Sons, vol. 103(1), pages 296-316, January.
    12. Donald Moynihan, 2018. "A great schism approaching? Towards a micro and macro public administration," Journal of Behavioral Public Administration, Center for Experimental and Behavioral Public Administration, vol. 1(1).
    13. Sunil Mithas & Yanzhen Chen & Yatang Lin & Alysson De Oliveira Silveira, 2022. "On the causality and plausibility of treatment effects in operations management research," Production and Operations Management, Production and Operations Management Society, vol. 31(12), pages 4558-4571, December.
    14. Alexander Ruder, 2019. "What Works at Scale? A Framework to Scale Up Workforce Development Programs," FRB Atlanta Community and Economic Development Discussion Paper 2019-1, Federal Reserve Bank of Atlanta.
    15. Yonatan Eyal, 2020. "Self-Assessment Variables as a Source of Information in the Evaluation of Intervention Programs: A Theoretical and Methodological Framework," SAGE Open, , vol. 10(1), pages 21582440198, January.
    16. Muller, Seán M., 2020. "The implications of a fundamental contradiction in advocating randomized trials for policy," World Development, Elsevier, vol. 127(C).
    17. Ralitza Dimova, 2019. "A Debate that Fatigues…: To Randomise or Not to Randomise; What’s the Real Question?," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 31(2), pages 163-168, April.
    18. Sophie van Huellen & Duo Qin, 2019. "Compulsory Schooling and Returns to Education: A Re-Examination," Econometrics, MDPI, vol. 7(3), pages 1-20, September.
    19. Justman, Moshe, 2018. "Randomized controlled trials informing public policy: Lessons from project STAR and class size reduction," European Journal of Political Economy, Elsevier, vol. 54(C), pages 167-174.
    20. John Gibson, 2019. "Are You Estimating the Right Thing? An Editor Reflects," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 41(3), pages 329-350.
    21. Moshe Justman, 2016. "Economic Research and Education Policy: Project STAR and Class Size Reduction," Melbourne Institute Working Paper Series wp2016n37, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:48:y:2024:i:4:p:579-609. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.