IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v48y2024i4p579-609.html

Randomized Controlled Trial Aversion among Public Sector Leadership: A Survey Experiment

Author

Listed:
  • Emily Cardon
  • Leonard Lopoo

Abstract

Background: While randomized controlled trials (RCTs) are typically considered the gold standard of program evaluation, they are infrequently chosen by public sector leaders, defined as government and nonprofit decision-makers, when an impact evaluation is required. Objectives : This study provides descriptive evidence on RCT aversion among public sector leaders and attempts to understand what factors affect their likelihood of choosing RCTs for impact evaluations. Research Design : The authors ask if public sector leaders follow similar preference patterns found among non-public sector leaders when choosing either an RCT or a quasi-experimental design and use a survey experiment to determine which factors affect the RCT choice. Subjects : The study sample includes 2050 public sector leaders and a comparison group of 2060 respondents who do not lead public sector organizations. Measures: The primary outcome measure is selecting an RCT as the preferred evaluation option. Results : When asked to make a decision about an impact evaluation, the majority of people do not choose an RCT. While also averse to RCTs, public sector leaders are about 13% more likely to prefer a RCT to a quasi-experimental evaluation compared to the general population. Public sector leaders are less likely to use RCTs for evaluations of more intense interventions, potentially because they are perceived to be superior to the options available for the control group. Conclusion : Funders should be aware that when given a choice, public sector leaders prefer other options to RCTs. Greater awareness of the benefits of RCTs could increase their use in the public sector.

Suggested Citation

  • Emily Cardon & Leonard Lopoo, 2024. "Randomized Controlled Trial Aversion among Public Sector Leadership: A Survey Experiment," Evaluation Review, , vol. 48(4), pages 579-609, August.
    • Handle: RePEc:sae:evarev:v:48:y:2024:i:4:p:579-609
    DOI: 10.1177/0193841X231193483
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X231193483
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X231193483?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Rachel Glennerster & Kudzai Takavarasha, 2013. "Running Randomized Evaluations: A Practical Guide," Economics Books, Princeton University Press, edition 1, number 10085, December.
    2. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    3. Weinstein, N.D. & Klotz, M.L. & Sandman, P.M., 1988. "Optimistic biases in public perceptions of the risk from radon," American Journal of Public Health, American Public Health Association, vol. 78(7), pages 796-800.
    4. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    5. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Florian Spitzer & Kira Abstiens & Sophie Karmasin, 2025. "Integrating behavioural insights in the policy process: on chances and hurdles identified by policy-makers and behavioural scientists," Mind & Society: Cognitive Studies in Economics and Social Sciences, Springer;Fondazione Rosselli, vol. 24(2), pages 621-663, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christopher J. Ruhm, 2019. "Shackling the Identification Police?," Southern Economic Journal, John Wiley & Sons, vol. 85(4), pages 1016-1026, April.
    2. Vellore Arthi & James Fenske, 2018. "Polygamy and child mortality: Historical and modern evidence from Nigeria’s Igbo," Review of Economics of the Household, Springer, vol. 16(1), pages 97-141, March.
    3. repec:wbk:wbrwps:10250 is not listed on IDEAS
    4. Vikram Dayal & Anand Murugesan, 2020. "Demystifying causal inference: ingredients of a recipe," IEG Working Papers 393, Institute of Economic Growth.
    5. John Cawley & Alex Susskind & Barton Willage, 2020. "The Impact of Information Disclosure on Consumer Behavior: Evidence from a Randomized Field Experiment of Calorie Labels on Restaurant Menus," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(4), pages 1020-1042, September.
    6. Florian F Gunsilius, 2025. "A primer on optimal transport for causal inference with observational data," Papers 2503.07811, arXiv.org, revised Mar 2025.
    7. Dina Pomeranz, 2017. "Impact Evaluation Methods in Public Economics," Public Finance Review, , vol. 45(1), pages 10-43, January.
    8. Jonathan Bauchet & Stacy Prieto & Jacob Ricker‐Gilbert, 2021. "Improved Drying and Storage Practices that Reduce Aflatoxins in Stored Maize: Experimental Evidence from Smallholders in Senegal," American Journal of Agricultural Economics, John Wiley & Sons, vol. 103(1), pages 296-316, January.
    9. Alexander Ruder, 2019. "What Works at Scale? A Framework to Scale Up Workforce Development Programs," FRB Atlanta Community and Economic Development Discussion Paper 2019-1, Federal Reserve Bank of Atlanta.
    10. Yonatan Eyal, 2020. "Self-Assessment Variables as a Source of Information in the Evaluation of Intervention Programs: A Theoretical and Methodological Framework," SAGE Open, , vol. 10(1), pages 21582440198, January.
    11. Sophie van Huellen & Duo Qin, 2019. "Compulsory Schooling and Returns to Education: A Re-Examination," Econometrics, MDPI, vol. 7(3), pages 1-20, September.
    12. Justman, Moshe, 2018. "Randomized controlled trials informing public policy: Lessons from project STAR and class size reduction," European Journal of Political Economy, Elsevier, vol. 54(C), pages 167-174.
    13. Moshe Justman, 2016. "Economic Research and Education Policy: Project STAR and Class Size Reduction," Melbourne Institute Working Paper Series wp2016n37, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.
    14. Seán M. Muller, 2021. "Evidence for a YETI? A Cautionary Tale from South Africa's Youth Employment Tax Incentive," Development and Change, International Institute of Social Studies, vol. 52(6), pages 1301-1342, November.
    15. Sarah Tahamont & Zubin Jelveh & Aaron Chalfin & Shi Yan & Benjamin Hansen, 2019. "Administrative Data Linking and Statistical Power Problems in Randomized Experiments," NBER Working Papers 25657, National Bureau of Economic Research, Inc.
    16. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    17. Sarah Tahamont & Zubin Jelveh & Aaron Chalfin & Shi Yan & Benjamin Hansen, 2021. "Dude, Where’s My Treatment Effect? Errors in Administrative Data Linking and the Destruction of Statistical Power in Randomized Experiments," Journal of Quantitative Criminology, Springer, vol. 37(3), pages 715-749, September.
    18. Margaret Dalziel, 2018. "Why are there (almost) no randomised controlled trial-based evaluations of business support programmes?," Humanities and Social Sciences Communications, Palgrave Macmillan, vol. 4(1), pages 1-9, December.
    19. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org, revised Apr 2025.
    20. Masahito AMBASHI & Naoyuki HARAOKA & Fukunari KIMURA & Yasuyuki SAWADA & Masakazu TOYODA & Shujiro URATA, 2025. "New Industrial Policies to Achieve Sustainable Asia-Wide Economic Development," Working Papers DP-2024-34, Economic Research Institute for ASEAN and East Asia (ERIA).
    21. Florent Bédécarrats & Isabelle Guérin & François Roubaud, 2019. "All that Glitters is not Gold. The Political Economy of Randomized Evaluations in Development," Development and Change, International Institute of Social Studies, vol. 50(3), pages 735-762, May.

    More about this item

    Keywords

    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:48:y:2024:i:4:p:579-609. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.